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Preface 


This  report  documents  the  activities  and  findings  of  Phase  I 
of  the  Squadron  Level  Training  (SLT)  research  program.  It 
specifically  examines  the  current  training  environment  at 
selected  operational  AF  units  {i.e.,  PACAF  F-16  maintenance 
units),  identifies  training  system  needs,  and  relates  them  to 
potential  technology  solutions.  This  report  is  the  culmination 
of  work  performed  by  Universal  Energy  Systems,  Inc.  and  the 
Armstrong  Laboratory  under  Task  50,  Contract  No.  F41689-86-D- 
0052. 
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Summary 


The  objective  of  the  Squadron  Level  Training  (SLT)  research 
program  is  to  examine  training  at  AF  operational  units  and 
determine  if  it  can  be  conducted  more  efficiently  and  effectively 
using  training  technologies.  A  number  of  trends  in  the  AF 
environment  prompted  concern  over  unit-level  training  and  led  to 
the  initiation  of  SLT  research.  These  trends  included  reduction 
in  force  size,  increased  skill  demands  placed  on  individuals, 
decreased  reliance  on  resident  training,  increased  complexity  of 
weapon  systems,  and  increased  training  responsibility  at  the  unit 
level.  This  paper  documents  the  conduct  and  findings  of  Phase  I 
of  a  two  phase  project.  The  goal  of  Phase  I  was  to  develop  an 
understanding  of  the  training  environment,  develop  a  preliminary 
approach  for  relating  training  system  needs  to  training 
technologies,  and  suggest  directions  for  continued  research.  In 
Phase  I,  researchers  developed  a  structured  interview  based  on 
the  major  functions  of  the  unit-level  training  system  including 
planning  and  programming,  development  and  delivery,  and 
management  and  evaluation  of  training.  They  administered  the 
interview  to  PACAF  F-16  flying  and  maintenance  units,  as  well  as 
personnel  at  higher  headquarters.  Content  analyses  of  the 
interview  data  revealed  a  number  of  general  training  problems  in 
the  unit-level  training  system.  Researchers  subsequently 
developed  an  ISD-based  framework  to  further  analyze  the  Phase  I 
findings  and  to  identify  potential  technological  solutions. 
Researchers  identified  the  specific  deficiencies  underlying  the 
general  training  problems  and  determined  that  AF  training 
technologies  addressed  each  of  the  deficiencies  to  some  degree. 


TRAINING  IN  PACAF  F-16  MAINTENANCE  UNITS: 

FINAL  REPORT  FOR  PHASE  I  OF  THE  SQUADRON 
LEVEL  TRAINING  RESEARCH  PROJECT 

I .  INTRODUCTION 

The  Air  Force  training  system  is  one  of  the  largest  and 
mostcomplex  training  systems  in  existence.  When  not  engaged  in 
combat, the  entire  Air  Force  can  be  viewed  as  "in  training”  to 
prepare  forwar.  Air  Training  Command  (ATC),  for  example, 
conducts  over  1400formal  technical  training  courses  as  well  as 
flying  training  (AFM  50-5,  1990;  C.  Yelverton,  HQ  ATC/ACC, 
personal  communication,  6  June,  1990).  Professional  military  and 
ancillary  training  is  also  conducted  at  various  levels  of  command 
throughout  the  Air  Force.  It  has  been  estimated  that,  during  any 
given  year,  at  least  half  of  the  total  Air  Force  is  involved  in 
some  aspect  of  training  ranging  from  short-term  skills  upgrade 
training  to  multi-year  professional  and  technical  development 
programs  (Johnson,  Green,  Soldwisch,  Turner,  &  Wall, 1988). 

Training  managers  at  all  levels  continually  weigh  many 
inputs  in  deciding  how  to  best  achieve  adequate  levels  of  job 
performance  to  ensure  mission  success.  Major  constraints 
management  must  deal  with  include  national  and  military  policy 
positions,  mission  requirements,  manpower  qualifications, 
funding,  and  systems  technology.  Frequently  one  or  more  of  these 
factors  can  undergo  rapid  change  or  redirection  which  require 
major  revisions  to  both  short-  and  long-term  training  goals.  The 
Air  Force  must  continually  adapt  and  restructure  its  training 
within  the  context  of  these  dynamic  constraints. 

Recent  international  and  domestic  trends  have  combined  to 
impact  on  the  factors  affecting  Air  Force  training  direction. 
Reductions  in  funding  and  force  size,  restructuring  of  career 
fields,  deployment  of  active  duty  and  reserve  personnel, 
increased  importance  of  on-the-job  training  (OJT),  changing  labor 
pool  demographics,  and  advances  in  technology  are  shaping  the 
answers  to  the  questions  about  the  "who,  what,  when,  where,  and 
how"  of  training.  In  the  future,  as  Air  Force  training  continues 
to  evolve,  it  is  anticipated  that  units  will  continue  to  be 
responsible  for  a  large  share  of  the  training  burden.  Since 
fewer  people  will  be  available  to  train  increasing  numbers  of 
tasks,  it  is  anticipated  that  training  technologies  will  be 
needed  at  the  unit  level  to  increase  the  efficiency  and 
effectiveness  of  training. 

The  Air  Force  Human  Systems  Division  (HSD),  in  its  role  as 
the  principal  advocate  for  the  human  operator  across  weapon  and 
support  systems,  conducts  programs  of  research  and  advances 
technology  to  prepare  the  human  operator  for  mission 
accomplishment.  In  response  to  a  commitment  at  all  levels  of  Air 
Force  command  to  increase  training  efficiency,  the  HSD  has  taken 
a  lead  role  in  the  investigation  of  how  technologies  can  be  used 
to  address  the  training  requirements  generated  by  an  increased 
emphasis  on  unit  training. 


1 


The  Human  Resources  Directorate  of  the  Armstrong  Laboratory 
(AL/HR)  is  supporting  HSD  by  identifying  and  evaluating  the  role 
of  training  at  the  unit  level  through  a  research  initiative 
referred  to  as  Squadron  Level  Training  (SLT).  (Note.  For  the 
purposes  of  this  project,  "squadron"  and  "unit"  generically  refer 
to  wing,  base,  squadron,  or  unit.  Figure  1  displays  the  types  of 
training  encompassed  by  the  term  "squadron  level  training."  A 
list  of  acronyms  and  terms  is  contained  in  the  Glossary  [Appendix 
A] . )  SLT  research  is  concerned  with  evaluating  the  potential  for 
utilizing  currently  available  and  planned  training  technologies 
at  the  unit  level.  The  primary  goal  of  the  SLT  research 
initiative  is  reflected  by  the  following  question:  Can  training 
done  at  the  operational  unit  be  done  more  effectively  and 
efficiently?  This  research  initiative  is  the  initial  step  toward 
understanding  the  requirements  of  SLT  and  the  potential  role  of 
HSD  technologies  in  enhancing  unit  training.  The  ultimate  goal 
of  this  research  stream  is  the  development  of  a  Science  and 
Technology  investment  strategy  for  continued  research  and 
development  (R&D)  of  training  technologies. 

The  objective  of  the  first  phase  of  SLT  was  to  assess 
qualitatively  the  state  of  training  in  selected  operational  AF 
units,  namely  F-16  aircraft  maintenance  units  of  the  Pacific  Air 
Force  (PACAF).  Following  is  a  list  of  training  issues  central  to 
Phase  I  of  the  study: 

1.  Trends  in  current  unit  training  requirements,  that  is, 
the  difference  between  a  trainee's  skills  and 
knowledges  and  those  required  by  the  job,  including 
type,  content,  and  amount  of  job-related  and  ancillary 
training  (e.g.,  number  of  trainees,  length  of  training, 
frequency  of  training ) ; 

2.  Existing  training  capabilities  of  the  unit,  that  is, 
the  resources  available  to  support  training  (e.g.,  time 
away  from  primary  duties,  materials,  equipment, 
experienced  personnel  to  serve  as  instructors ) ; 

3.  Identification  of  factors  which  facilitate  or  impede 
unit-level  training  (e.g.,  anticipated  impact 

of  increased  unit-level  training). 

4 .  Current  training  system  needs  and  opportunities  for 
increasing  the  efficiency  and  effectiveness  of  training 
at  the  operational  unit,  including  identification  of 
technologies  needed  to  support  training; 

Outcomes  from  this  phase  of  the  SLT  investigation  will  be 
evaluated  to  determine  the  necessity  of  further  research.  It  is 
hoped  that  this  preliminary  research  will  provide  a  context  and 
suggest  directions  for  a  more  rigorous  and  systematic  analysis  of 
AF  unit  level  training  in  a  subsequent  phase. 
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Figure  1.  Depiction  of  the  Squadron  Level  Training  Domain 


This  report,  then,  documents  the  conduct  and  findings  of 
the  first  phase  of  SLT  research  in  the  maintenance  field.  The 
following  sections  present  the  research  methodology,  results,  and 
conclusions  of  the  research  with  regard  to  the  training  issues 
cited  above.  Additionally,  this  report  describes  related, 
concurrent  efforts  to  develop  a  training  needs  -  technology 
matching  procedure  and  apply  it  to  Phase  I  findings. 


II.  RESEARCH  METHODOLOGY 
A.  Data  Collection 

The  data  collection  for  this  research  effort  was  conducted 
in  two  general  stages:  first,  informational  interviews  were 
conducted  at  the  headquarters  level  and,  then,  structured 
interviews  were  conducted  at  the  base  level.  Figure  2  presents 
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the  general  sampling  scheme  for  identifying  relevant  sources  of 
information  required  to  support  this  task. 

Training  personnel  at  the  headquarters  level  were 
interviewedfor  information  relevant  to  training  issues,  for 
background  information  gathering,  and  instrument  development 
purposes.  These  sources  are  identified  in  Figure  2  as  being  a 
part  of  the  "Collateral"  data  collection.  A  list  of  the  specific 
offices  contacted  in  this  stage  is  contained  in  Appendix  B. 
Archival  data  (e.g.,  manning  documents,  training  records,  IG 
reports)  and  reference  documents  (e.g.,  regulations,  briefings, 
talking  papers)  were  collected  from  these  sources  as  appropriate. 

The  actual  field  data  collection  took  place  at  three 
Pacific  Air  Force  (PACAF)  bases:  Kunsan  AB,  Korea;  Osan  AB, 

Korea;  and  Misawa  AB, Japan.  Group  and  individual  interviews  were 
conducted  over  a  three  week  period  from  27  August  to  14 
September,  1990  (Table  1).  This  series  of  interviews  is 
referenced  in  Figure  2  as  "Primary"  data  collection. 

Interviewers  were  two  officers  from  the  AL/HRTE  and  two 
researchers  from  UES,  each  of  whom  had  been  involved  in  the 
initial  information  gathering  and  instrument  development 
activities. 

All  data  from  the  maintenance  portion  of  the  first  phase  of 
the  SLT  research  initiative  have  been  collected  and  analyzed. 

The  following  section  provides  details  related  to  the  conduct  of 
field  interviews,  preparation  of  the  data,  and  analysis 
approaches . 

Subjects 

Data  collection  focused  on  the  three  primary  maintenance 
units  comprising  the  Combat  Oriented  Maintenance  Organization 
(COMO)  of  PACAF:  Component  Repair  Squadron  (CRS),  Aircraft 
Generation  Squadron  (AGS),  and  Equipment  Maintenance  Squadron 
(EMS).  Messages  were  sent  by  the  AL/HR  to  a  point-of-contact  in 
the  Training  Management  Division  (TMD)  at  each  base  several  weeks 
prior  to  arrival  of  the  interview  team.  Messages  requested  that 
individuals  with  direct  experience  in  one  or  more  of  the  three 
general  training  functions  be  scheduled  for  interviews  (i.e., 
Planning/Programming,  Management /Evaluation, 

Development /Delivery ) .  Suggested  personnel  included:  TMD  (i.e., 
MA,  MAT),  Quality  Control  (MAQ),  squadron  supervisors, 
squadron/shop  training  monitor,  supervisors/OJT  trainers, 
mechanics/specialists,  base  OJT,  and  Field  Training  Detachment 
(FTD)  supervisors /instructors.  Additionally,  several  enlisted 
specialties  were  identified  as  being  desired  candidates  for 
interviewing  (i.e..  Air  Force  Specialty  Codes  (AFSCs]  454X1, 
324X0,  452X4,  461X0).  Data  were  collected  from  22  enlisted 
specialties  and  Air  Force  Engineering  and  Technical  Services 
(AFETS)  civilians.  (Data  collection  also  included  two  officer 
specialties,  but  this  was  considered  incidental  to  the  major  data 
collection  effort).  Table  2  displays  the  sample  demographics. 
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Table  1 


Participation  bv  Data  Collection  Site 


Number  of  Interview  Participants 
Group  Individual  Total  by 

Base 

Base _ Sessions  (N) _ Sessions  (N) _ Sessions  (N) 


Misawa 

33 

(107) 

14 

(14) 

47 

(121) 

Kunsan 

9 

(23) 

40 

(40) 

49 

(63) 

Osan 

11 

(31) 

19 

(19) 

30 

(50) 

53 

(161) 

73 

(73) 

126 

(234) 

Total  Sessions  =  126 

Total  Participants  (N)  =  234 


Interview  Protocol 


A  comprehensive  interview  protocol  was  developed  for  use  in 
this  project  (Appendix  C).  The  survey  was  designed  to  be 
applicable  across  all  training  functions  with  independent 
sections  focused  on  the  operational  perspective  as  well  as  the 
areas  of  Planning/Programming,  Management /Evaluation,  and 
Development /Delivery .  Thus,  the  interview  protocol  could  be 
tailored  for  each  interviewee  dependent  on  their  role  in  the 
training  process.  The  survey  consisted  primarily  of  open-ended 
and  Yes/No  questions,  although  rating  scales  were  included  in 
portions  of  the  survey  (e.g.,  task  criticality  ratings, 
evaluations ) . 

Procedure 

All  participants  were  given  a  brief  introduction  to  the  SLT 
project  that  described  the  purposes  of  the  study  and  their 
personalinvolvement .  Then,  participants  completed  a  brief 
demographic  sheet;  requested  information  included  grade,  current 
AFSC,  current  position,  length  of  time  in  unit  and  position, 
total  maintenance  time,  total  F-16  maintenance  time,  current  role 
in  unit  training  process,  major  command  (MAJCOM),  base,  and  unit. 

All  interviews  were  conducted  using  the  survey  as  a 
framework,  although  deviations  were  made  as  needed  to  accommodate 
time  considerations,  extent  of  interviewees'  involvement  in  the 
training  process,  responsiveness  of  the  interviewees,  and  so  on. 
Interview  sessions  typically  required  90  minutes,  although  some 
ran  longer  and  others  were  cut  short  of  completion  due  to  mission 
demands  or  scheduling  problems. 
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B.  Analysis 


Types  of  Data 

Three  major  types  of  data  were  included  in  the  analyses: 

(a)  interviewee  demographic  data,  (b)  coded  responses  to 
interview  questions,  and  (c)  interview  comments  which  were  not 
deemed  suitable  for  coding.  All  of  these  contributed  to  the 
analysis  phase  of  this  project.  Additionally,  documents  (e.g., 
training  regulations,  briefing  slides,  talking  papers)  and 
archival  data  (e.g.,  manning  documents.  Training  Quality  Reports 
[TQRs],  inspection  reports,  OJT  training  records  and  status 
reports)  served  as  background  resources  and  reference  materials 
for  analysts. 

Data  Preparation  and  Analysis 

Interviewers'  notes  served  as  the  basis  for  all  content 
analyses.  Upon  completion  of  all  interviews  and  return  from  data 
collection  travel,  these  notes  were  transcribed  to  enhance 
readability  and  thoroughness.  These  were  later  reviewed  and 
coding  schemes  were  developed  to  capture  the  variety  of  responses 
to  questions.  Upon  finalization  of  coding  schemes,  the  interview 
notes  were  re-read  and  codes  were  assigned  to  each  item  as 
appropriate.  More  detail  on  this  process  is  given  below. 

Content  Analyses.  Relevant  Air  Force  training  needs 
analysis  research  projects  were  identified  and  reviewed  for 
details  on  the  qualitative  analyses  employed  (e.g.,  Bruce, 
Rockway,  &  Povenmire,  1989;  Carson,  Chambers,  Gosc,  1984; 

Johnson,  Damewood,  &  Phelps,  1987;  Wilson  &  Comander,  1989). 

These  reports  give  little  or  no  information  about  analysis 
techniques  used  for  evaluation  and  summarization  of  open-ended 
interview  data.  Other  sources,  however,  provided  guidance  for 
the  qualitative  analyses. 

Content  analysis  is,  by  definition,  "The  systematic 
assessment  of  the  manifest  content  of  communications.  The 
assessment  involves  standardized  procedures  for  the  analysis  of 
written,  oral,  musical,  or  gestural  communications.  Content 
analyses  are  useful  for  research  if  for  some  reason  it  is 
impossible  to  obtain  opportunities  for  direct  observations" 
(Yaremko,  Harari,  Harrison,  &  Lynn,  1982,  p.  40).  Since  it  was 
not  feasible  for  researchers  in  Phase  I  of  the  SLT  research  study 
to  gather  information  through  direct  observation  of  training,  the 
collection  of  data  through  the  use  of  interview  protocols  and 
archival  research  was  appropriate,  and  content  analysis  is  a 
suitable  evaluation  and  summarization  approach. 
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Table  2 

Sample  Demographic  Data 


Category 

Frequency 

(%  of 

Total ) 

AGS 

UNIT 

51 

(22%) 

CRS 

70 

(30%) 

EM  S 

82 

(35%) 

FTD 

4 

(2%) 

MA 

3 

(1%) 

MAT 

16 

(7%) 

QA 

4 

(2%) 

Wing 

4 

(2%) 

El 

GRADE 

9 

(4%) 

E2 

20 

(9%) 

E3 

8 

(8%) 

E4 

35 

(15%) 

E5 

75 

(32%) 

E6 

39 

(17%) 

E7 

28 

(12%) 

E9 

1 

(<1%) 

03 

1 

(<1%) 

04 

3 

(1%) 

AFETS 

5 

(2%) 

CURRENT 

FTD  Instructor 

TRAINING  ROLE 

4 

(2%) 

MAT  Instructor 

6 

(3%) 

MAT  Training  Manager 

5 

(2%) 

ISD  Technician 

2 

(1%) 

Squadron  Training  Monitor 

6 

(3%) 

Squadron  Commander/Supervii 

3or 

35 

(15%) 

Training  Coordinator 

24 

(10%) 

OJT  Trainer 

57 

(24%) 

OJT  Trainee 

39 

(17%) 

AFETS 

5 

(2%) 

MAT  Supervisor 

6 

(3%) 

Base  OJT 

2 

(1%) 

None  of  the  Above 

43 

(18%) 
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(Table  2  Cont.) 


Category 


Frequency  (%  of  Total) 


CURRENT  AFSC 
Civilian 


AFETS 

Officer 

4016  Maintenance  Staff  Officer 

4024  Aircraft  Maintenance  Officer 

Enlisted 

324X0  Precision  Measurement  Equipment  Laboratory 

45XXXa  Manned  Aerospace  Maintenance 

452X2  Avionics  Systems  (F-16) 

452X4  Crew  Chief 

452X5  Tactical  Electronic  &  Environmental  Systems 

452XXa  Avionics  Systems 

454X0  Jet  Engine 

454X1  Aerospace  Ground  Equipment 

454X2  Aircrew  Egress  Systems 

454X3  Aircraft  Fuel  Systems 

454X4  Aircraft  Pneudraulic  Systems 

454X5  rategic  Electronic  &  Environmental  Systems 

455X0  Ihoto  &  Sensors  Maintenance 

455X5  Avionics  Support  Equipment 

456X1  Electronic  Warfare  Systems 

458X0  Aircraft  Metals  Technology 

458X1  Nondestructive  Inspection 

458X2  Aircraft  Structural  Maintenance 

458XXa  Aircraft  Maintenance 

461X0  Munitions  Systems 

462X0  Aircraft  Armament  Systems 

751X0  Education 

751X1  Training  Systems 

75lXXa  Education  and  Training 


5 

(2%) 

3 

(1%) 

1 

(<1%) 

9 

(4%) 

3 

(1%) 

18 

(8%) 

33 

(14%) 

11 

(5%) 

2 

(<1%) 

18 

(8%) 

16 

(7%) 

8 

(3%) 

7 

(3%) 

3 

(1%) 

1 

(<1%) 

4 

(2%) 

1 

(<1%) 

10 

(4%) 

4 

(2%) 

2 

(1%) 

12 

(5%) 

1 

(<1%) 

19 

(8%) 

30 

(13%) 

1 

(<1%) 

10 

(4%) 

2 

(<1%) 

a 

Incomplete  specialty  code  provided  by  participants. 
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(Table  2  Cont. ) 


Tenure  of  ParticiDants 

( in  Months) 

Mean 

SD 

Range 

Median 

Time  in  Unit 

14.6 

12.9 

1-72 

10 

Time  in  Position 

12.0 

11.1 

1-72 

9 

Total  Maintenance  Time 

101.6 

75.1 

0-374 

96 

F-16  Maintenance  Time 

33.6 

34.1 

0-180 

21 

Pf af fenberger  (1988)  described  three  activities  required  in 
qualitative  analysis.  Each  of  these  were  accomplished  during  the 
SLT  research  analysis  phase  and  are  briefly  noted  here. 


1.  Rewriting.  The  fast  pace  of  most  field  interviews 
requires  that  interviewers  review  their  notes  and  rewrite 
findings  in  more  detail.  Pfaf fenberger  (1988)  cited  that 
the  retrospective  rewriting  of  notes  is  an  integral  part  of 
field  research  which  does  more  than  stimulate  memory  and 
stated  that  "Such  acts  of  rewriting  are  an  important  form 
of  data  analysis  and  theoretical  discovery  in  themselves" 
(p.  26).  This  recapture  of  information  helps  researchers 
to  provide  the  context  the  information  which  may  be  missing 
from  that  actual  statements  made  by  interviewees. 

2.  Coding.  This  activity  is  the  process  of  attaching 
category  names  or  labels  to  the  basic  units  of  field 
research  data,  in  this  case,  responses  to  open-ended  survey 
questions.  The  development  of  categories  is  an  on-going 
process  with  modifications  occurring  as  needed  to  best 
describe  the  responses.  Pfaf fenberger  (1988)  listed  four 
relevant  strategies  for  coding  SLT  data:  (a)  let  coding 
categories  emerge  from  the  data;  (b)  develop  general, 
abstract  categories  that  fit  the  data;  (c)  classify  data 
and  develop  typologies;  and  (d)  change  and  refine  the 
categories  as  understanding  improves.  The  first  and  last 
of  these  approaches  were  used  by  analysts  for  this  project. 

3.  Comparison.  In  this  stage  of  analysis,  the  responses 
are  analyzed  in  accordance  with  the  associated  categories; 
responses  can  be  tallied  and  quantitatively  analyzed.  In 
this  manner,  consistent  remarks  among  interviewees  can  be 
recorded  according  to  a  coding  scheme.  For  those  questions 
to  which  responses  cannot  be  coded,  content  can  be  compared 
across  interviewees  for  unique  individual  contributions  or 
general  agreement. 


Responses  to  SLT  interview  questions  were  reviewed  by 
analysts  and,  where  possible,  coding  schemes  were  developed  and 
employed  to  comprehensively  categorize  and  quantify  classes  of 


responses.  As  an  example  of  the  comprehensiveness  of  this 
effort,  71  different  types  of  training  were  noted  by  interviewees 
and  coded  by  analysts.  In  this  manner,  the  open-ended  responses 
were  labeled  and  tallied  for  quantified  analysis  described  below 
(e.g.,  descriptive  statistics). 

Phase  I  Analyses.  The  content  analysis  efforts  during  this 
phase  of  research  were  aimed  at  identifying  "the  manifest 
content"  of  the  interview  data.  The  training  issues  central  to 
this  research  effort  formed  the  basic  structure  for  the  analysis. 
They  included:  1)  discerning  important  trends  in  unit-level 
training  requirements;  2)  evaluating  current  unit  training 
capabilities;  3)  identifying  factors  facilitating  or  impeding 
training;  and,  4)  identifying  training  system  needs  and 
opportunities  for  technology  application.  Particular  attention 
was  paid  to  these  issues  as  they  related  to  the  instructional 
system  development  process,  from  the  analysis  of  job  and  training 
requirements  to  the  development,  delivery,  and  evaluation  of 
training . 

As  mentioned  above,  coding  schemes  were  developed  for  as 
many  of  the  interview  items  deemed  appropriate.  After  coding  of 
all  interviews  had  been  accomplished,  the  demographic  data  and 
item  responses  were  entered  into  a  database  consisting  of  one 
record  per  participant.  Each  case  was  entered  twice  and  compared 
for  100  percent  verification  of  accuracy. 

Frequency  analyses  were  conducted  on  categorical  response 
items  and  demographics  questions.  Descriptive  statistics  (e.g., 
mean,  standard  deviation,  mode,  range)  were  conducted  on  all 
continuous  data  fields.  Where  appropriate,  the  sample  was 
divided  into  groups  according  to  one  or  more  demographic 
variables  (e.g.,  unit,  AFSC)  to  allow  for  examination  of 
responses  across  groups. 

As  will  be  noted  in  the  results  discussed  in  the  following 
sections,  missing  data  and/or  interviewee  failures  to  give 
responses  were  frequent  occurrences  in  this  study,  thereby 
restricting  the  sample  size  for  most  analyses.  A  number  of 
reasons  can  be  given  for  this.  In  many  instances,  for  example, 
an  individual  was  not  asked  a  particular  question  dependent  on 
their  responses  to  another  question.  Thus,  these  are  not 
"missing,"  but  rather,  "not  appropriate."  Also,  interviewees 
may  have  been  unresponsive  due  to  factors  such  as  apathy  or  non¬ 
applicability.  Other  primary  causes  of  missing  data  were 
incomplete  interview  sessions  because  of  scheduling  problems, 
interruptions  due  to  mission  activity  or  natural  disasters  (i.e., 
flooding  at  Osan  AB),  and  omissions  by  the  interviewer. 
Consequently,  quantitative  results  should  be  interpreted 
cautiously  due  to  the  "missing  data."  For  each  set  of  results 
presented  here,  the  total  number  of  respondents  is  noted. 

As  shown  in  Table  1,  73  individual  and  53  group  interviews 
were  conducted.  For  data  analysis  purposes,  the  comments  of  each 
group  interview  were  aggregated  and  counted  as  a  single  response. 
Thus,  a  group  of  three  is  represented  by  a  single  data  point,  a 
group  of  five  is  counted  as  a  single  data  point,  and  so  on.  In 


this  manner,  the  group  and  individual  interviews  were  counted 
equally  in  the  descriptive  statistics.  In  the  analyses  and 
reporting  of  interview  responses,  the  total  sample  size  was  126 
(i.e.,  73  individuals  and  53  groups).  Demographic  data  were, 
however,  collected  from  each  participant  and  the  figures  reported 
in  Table  2  reflect  the  entire  sample  of  personnel  interviewed  (N 
-234). 


III.  TRAINING  ISSUES 

This  chapter  discusses  the  apparent  trends  in  current 
training  requirements,  current  training  capabilities,  and  factors 
facilitating  or  impeding  the  training  system  in  general.  The 
final  section  in  this  chapter  briefly  summarizes  the  major 
training  problems  which  have  surfaced  during  our  analyses  of 
training  issues. 

It  should  be  noted  that  these  results  represent  the 
subjective  conclusions  of  our  analysts  based  on  an  analysis  of 
interview  content  and  related  archival  data.  They  were  based  on 
data  collected  at  F-16  maintenance  units  in  PACAF  and,  therefore, 
should  be  interpreted  as  such.  Appendix  D  contains  summary 
tables  which  support  the  discussion  of  the  training  issues;  these 
are  referenced  here  as  appropriate.  Subsequent  chapters  address 
potential  application  of  training  technologies  for  the  effective 
resolution  of  these  issues. 

A.  Trends  in  Current  Training  Requirements 

Air  Force  Manual  50-2  (1986)  defines  training  requirements 
as  "...  those  skills  and  knowledges  which  are  required  for 
satisfying  the  job  performance  requirements,  and  not  already  in 
the  students'  incoming  repertoire"  (p.  77).  By  definition,  a 
training  requirement  exists  when  a  deficit  exists  between  an 
individual '8  acquired  skills  and  knowledges  and  the  job 
performance  requirements.  The  findings  of  Phase  I  indicate  there 
are  at  least  three  significant  and  consistent  drivers  of  training 
requirements  in  the  maintenance  communities  of  the  three  bases 
sampled.  Training  requirements  tend  to  be  driven  most  frequently 
by:  (a)  experience  levels,  (b)  technology,  and  (c)  specialty 
restructuring  through  Rivet  Workforce.  The  following  sections 
will  be  devoted  to  an  examination  of  these  drivers  and  trends. 

Due  to  the  wide  variety  of  responses,  no  attempts  will  made  to 
define  AFSC-specific  requirements. 

Experience  Levels  and  Training  Requirements 

Identification  of  prior  experience  and  training  in  the 
weapon  system  is  a  precursor  to  determining  training 
requirements.  Data  analyses  indicate  that  91%  (68  of  75)  of  the 
maintenance  supervisors  interviewed  determined  adequacy  of 
experience  by  conducting  an  initial  interview  with  all  new 
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arrivals  (Appendix  D,  Table  D-l).  The  interview  generally 
consisted  of  a  review  and  discussion  of  the  individual's 
documented  job  experience  and  training.  Over-the-shoulder 
evaluations  on  selected  tasks  were  also  performed  by  35%  (26  of 
75)  of  the  supervisors  as  a  part  of  this  experience  assessment 
process.  A  training  requirement  generally  exists  when  the 
supervisor  detects  a  skill  and/or  knowledge  deficit. 

When  asked  about  the  current  trends  in  experience  levels  of 
maintainers,  52%  (60  of  115)  of  the  respondents  felt  experience 
levels  were  on  the  decline  (Appendix  D,  Table  D-2 ) .  Lack  of 
experience  of  incoming  personnel  was  identified  by  interviewees 
as  the  most  significant  driver  of  training  requirements.  All 
newly  assigned  maintainers  in  PACAF  were  required  by  PACAF 
Regulation  50-17  (1989)  to  receive  specific  maintenance  training 
upon  arrival  at  the  unit  (e.g.,  general  maintenance  orientation 
which  focuses  on  local  policies ,  procedures,  requirements, 
conditions,  and  so  on).  Only  those  lacking  adequate  or  current 
job-related  experience  (i.e.,  have  not  worked  on  the  systems  in 
the  last  five  years)  normally  required  additional  skill  training 
before  they  were  permitted  to  perform  unsupervised  and  unassisted 
tasks. 

Interview  findings  revealed  that  the  primary  sources  of 
experience-driven  training  requirements  were  newly  assigned 
technical  school  graduates  and  maintainers  assigned  from  other 
weapon  systems.  Of  the  50  interviewees  who  identified  factors 
contributing  to  a  perceived  decline  in  experience,  29  (58%) 
blamed  turnover  for  the  decline  (i.e.,  experienced  maintainers 
leaving  and  being  replaced  by  inexperienced  people),  while  15 
(30%)  specified  too  much  cross-command/weapon  system  movement  as 
the  culprit. 

Although  many  (37%,  42  of  115)  of  the  supervisors  and 
trainers  interviewed  believed  technical  schools  were  generally 
doing  an  adequate  job  of  training,  they  contended  3-skill  levels 
lacked  some  basic  maintenance  skills  (e.g.,  hands-on  experience) 
and  required  extensive  training  before  they  were  productive  on 
the  job  (Appendix  D,  Table  D-3).  Likewise,  maintainers  with 
experience  in  other  systems  also  required  training  to  become 
familiar  with  the  systems  in  use  at  the  unit.  However,  the 
amount  of  training  is  significantly  less  than  that  required  for 
3-skill  level  personnel  because  the  experienced  maintainer  (i.e., 
5-  or  7-skill  level)  has  usually  acquired  skills  which  were 
readily  transferable  to  the  new  system.  Table  3  presents  data 
from  6  AFSCs  on  the  time  required  for  3-skill  levels  upgrading  to 
5-skill  level  and  time  for  newly  assigned  experienced  maintainers 
to  become  position  qualified.  Mote  that  the  incoming  3-skill 
level  personnel  required  a  considerable  amount  of  time  to  become 
position-qualified  compared  to  those  with  appropriate  experience. 
The  impact  of  lack  of  experience  on  incoming  personnel  at  short 
tour  assignments  is  discussed  later  in  this  section. 
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Technolo 


av  and  Training  Requirements 

Technological  advances  in  operational  systems  have  been 
both  beneficial  and  problematic  for  the  maintenance  community. 
System  sophistication,  in  addition  to  creating  weapon  systems 
with  increased  probabilities  for  mission  success  and  increased 
survivability,  has  improved  system  maintainability.  Many  of  the 
systems  on  the  F-16,  for  example,  consist  of  modular,  "black  box" 
technologies.  In  addition,  many  systems  are  now  self-diagnostic 
such  that  the  system  identifies  malfunctioning  components.  For 
the  maintenance  technician,  those  features  translate  to  less 
labor-intensive  routine  maintenance  as  well  as  rapid,  relatively 
easy  identification  and  repair  of  system  malfunctions.  Because 
of  this  sophistication,  much  of  the  maintenance  on 
state-of-the-art  equipment  has  been  reduced  to  simple  removal  and 
replacement  of  components  which  are  sent  to  the  repair  shop  or 
thrown  away. 


Table  3 

Average  Time  fin  Months)  Reported  for  Incoming  Personnel  to 
Become  Position  Qualified 


3-Skill 

Level 

ExDerienced 

T-Test 

AFS 

Mean 

SD 

N 

Mean 

SD 

N 

t 

df 

p 

452X2 

11.0 

1.0 

3 

3.0 

2.8 

2 

. 

. 

(Avionics) 

452X4 

8.8 

2.9 

10 

4.3 

3.5 

8 

3.0 

5  . 

031 

(Crew  Chief) 

454X0 

8.8 

2.6 

5 

4.7 

1.5 

3 

7.0 

1  . 

090 

(Jet  Engine) 
454X1 

9.6 

3.3 

5 

0.3 

0.8 

6 

6.8 

4  . 

002 

(AGE) 

461X0 

7.7 

2.4 

6 

1.4 

1.1 

5 

6.2 

4  . 

003 

(Munitions ) 
462X0 

9.2 

3.0 

10 

4.4 

1.7 

5 

3.0 

4  . 

040 

(Armament ) 

Entire  Sample 

9.1 

2.8 

71 

3.6 

2.9 

48 

11.0 

37  . 

000 

Note.  Paired  t-test  procedure  reduced  sample  to  1  case  for  AFS 
452X2,  preventing  statistical  analysis. 
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Although  technological  advancement  has  been  largely 
beneficial  for  maintenance  activities,  it  has  also  created  some 
unique  challenges  in  the  area  of  maintenance  training.  For 
example,  while  increased  reliability  of  equipment  reduces  some  of 
the  maintenance  requirements  (e.g.,  increased  mean  time  between 
failure ) ,  it  limits  the  opportunity  for  OJT  of  certain  tasks  on 
actual  equipment.  Because  of  the  heavy  reliance  on  hands-on  OJT 
throughout  the  maintenance  community  and  the  generally  accepted 
policy  of  not  "breaking”  equipment  just  to  train,  the  opportunity 
to  train  on  actual  equipment  decreases  as  the  reliability  of 
systems  increase.;. 

Regardless  of  whether  the  maintainers '  job  becomes  simpler 
or  more  complex,  from  a  training  perspective,  changes  in 
technology  usually  mean  additional  or  new  training  requirements. 
New  terms,  different  procedures,  and  specialized  equipment  and 
tools  must  be  learned. 

Rivet  Workforce  and  Training  Requirements 

Rivet  Workforce  (RWF)  is  the  designation  for  an  Air 
Force-wide  initiative  to  restructure  aircraft  maintenance 
specialties.  The  objective  of  RWF  is  to  create  a  more  flexible 
maintenance  force  by  broadening  the  skills,  knowledge,  and 
experience  of  aircraft  maintenance  personnel  in  selected 
specialties.  Typically,  RWF  combined  two  or  more  specialties 
into  one  specialty,  or  it  eliminated  specialization  within  a 
specialty  ( i . e . ,  shredouts ) .  Implementation  of  RWF  initiated  a 
period  of  dramatically  increased  training  requirements  in  those 
specialties  affected  by  the  program.  During  this  transition 
period,  all  affected  maintainers  receive  training  and  must  become 
qualified  to  perform  the  additional  tasks  in  their  new  specialty. 
This  transition  process  has  been  called  "rivetization"  by  the 
maintenance  community. 

Perhaps  the  best  example  of  the  impact  of  RWF  on  training 
requirements  identified  during  SLT  data  collection  was  in  the  new 
F-16  Tactical  Aircraft  Maintenance  specialty  (AFSC  452X4B). 
Maintainers  from  three  separate  specialties  (i.e.,  crew  chief, 
flightline  jet  engine,  and  hydraulics)  were  combined  into  the  one 
specialty.  The  "rivetization"  of  maintainers  in  AFSC  452X4B 
created  additional  training  requirements  in  formal  training  as 
well  as  OJT.  In  essence,  each  person  in  the  new  AFSC  must  have 
acquired  all  of  the  required  skills  of  the  other  two  specialties. 
Rivetization  has  been  particularly  problematic  at  the  data 
collection  sites  where  training  capability  and  resources, 
including  time,  are  extremely  limited.  This  situation  was 
further  aggravated  by  the  fact  that  the  training  requirements 
themselves  were  unclear.  The  MAT  at  Kunsan  AB  was  constructing 
their  own  Job  Qualification  Standards  (JQSs)  and  training 
requirements  lists  because  of  this.  RWF  was  also  cited  as  a 
factor  in  the  decline  in  the  experience  levels  of  maintenance 
personnel  because  it  has  diluted  experience  levels  in  some 
specialties  (e.g.,  AFSC  452X4B  [F-16  Tactical  Aircraft 


15 


Maintenance],  AFSC  452X5  [Tactical  Electrical  and  Environmental 
Systems])  (Appendix  D,  Table  D-2).  According  to  some 
technicians,  RWF  reduced  the  credibility  of  Quality  Assurance 
(QA)  evaluations  because  QA  personnel  had  to  evaluate  rivetized 
tasks  with  which  they  were  not  entirely  familiar. 

B.  Evaluation  of  Current  Training  Capabilities 

PACAF  aircraft  maintenance  training  relies  on  an  extensive, 
multi-faceted  approach.  Virtually  every  enlisted  maintainer, 
regardless  of  rank,  skill  level,  or  experience,  receives 
job-related  task  or  ancillary  training  as  a  result  of  his/her 
assignment  to  the  command.  The  key  elements  of  the  PACAF 
aircraft  maintenance  training  program  are  (a)  enroute  training, 

(b)  the  Aircraft  Maintenance  Qualification  Program  (AMQP) ,  and 

(c)  OJT.  This  section  will  first  discuss  the  contribution  of 
each  of  these  training  settings  to  PACAF  squadron-level 
maintenance  training,  and  subsequently,  focus  on  the  resources 
which  were  available  to  support  each  aspect  of  this  training 
system  ( i . e . ,  planning ,  programming ,  management ,  evaluation , 
development  and  delivery).  Refer  to  Figure  1  for  a  depiction  of 
the  current  training  structure. 

The  objective  of  the  PACAF  aircraft  maintenance  program  is 
"to  develop  and  apply  cost  effective  training  which  teaches  job 
essential  skills  and  knowledge"  (PACAFR  50-17,  1989). 
Additionally,  because  many  of  the  assignments  in  PACAF  are  short 
tour  (i.e.,  12  months),  the  command  goal  is  to  provide  as  much 
required  mission-essential  technical  and  ancillary  training  as 
possible  prior  to  an  individual's  arrival  at  the  unit.  A  total 
of  2961  enlisted  maintenance  personnel  in  the  45  (Manned 
Aerospace  Maintenance)  and  46  (Munitions  and  Weapons)  functional 
communities  were  assigned  to  the  maintenance  complexes  of  the 
three  data  collection  bases.  This  number  approximates  the  annual 
trained  personnel  requirement  (TPR)  for  those  bases.  The 
training  programs  discussed  here  must  possess  the  capability  to 
train  on  an  annual  basis  the  vast  majority  of  these  personnel 
since  most  will  be  reassigned  each  year. 

Training  Settings 

Enroute  Training.  Enroute  training  occurs  in  conjunction 
with  a  permanent  change  of  station  (PCS)  assignment,  but  prior  to 
arrival  at  the  new  duty  station.  It  is  intended  to  ensure  that 
maintenance  personnel  arrive  at  their  units  with  the 
mission-essential  skills  needed  to  be  immediately  productive. 
Enroute  training  is  especially  critical  for  those  who  lack 
experience  in  the  systems  they  will  maintain  at  their  new 
assignment.  Technical  school  graduates,  personnel  from  other 
commands,  and  maintainers  who  lack  recent  experience  in  the 
system  (i.e.,  within  the  last  five  years)  are  those  for  whom 
enroute  training  was  designed. 
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Enroute  training  requirements  are  identified  through  a 
coordinated  effort  between  the  personnel,  maintenance,  training, 
and  operational  activities  involved  in  assignment  of  the 
maintainer  to  PACAF.  When  an  individual  is  selected  for 
assignment  to  a  PACAF  unit,  appropriate  notifications  are  routed 
to  the  gaining  unit  through  the  personnel  and  maintenance  staffs 
at  both  HQ  PACAF  and  the  base.  The  gaining  unit  reviews  the 
documented  experience  and  training  of  the  individual  being 
reassigned  and  determines  whether  the  individual  requires  enroute 
training  to  do  the  assigned  job.  If  training  is  required,  the 
process  is  essentially  reversed,  becoming  a  request  to  ATC  to 
provide  enroute  training  at  an  appropriate  FTD.  (Mote.  A 
detailed  description  of  this  process,  including  the  office 
symbols  of  participants  and  administrative  procedures,  can  be 
found  in  Chapter  5,  PACAFR  50-17.)  ATC's  3785th  Field  Training 
Wing,  headquartered  at  Sheppard  AFB,  TX,  is  responsible  for 
conducting  enroute  training  at  its  FTDs. 

Interview  findings  indicate  that  maintenance  supervisors 
were  becoming  increasingly  concerned  about  the  number  of  people 
arriving  who  did  not  receive  the  needed  enroute  training.  HQ 
PACAF/LGMMR  staff  admitted  that  more  people  were  assigned  without 
receiving  enroute  training  than  they  would  like  to  have  seen. 

They  blamed  part  of  the  problem  on  communications  delays  in  the 
assignment  notification,  unit  review,  and  training  request 
process.  Several  interviewees  at  HQ  PACAF  suggested  that 
electronic  mail  would  greatly  enhance  their  ability  to  speed  up 
the  process  and  ensure  most,  if  not  all,  receive  the  required 
enroute  training. 

Lack  of  enroute  training  was  of  particular  concern  at  the 
Korean  bases  because  of  a  lack  of  adequate  FTD;  Kunsan  AB  has  no 
FTD  and  Osan  AB  has  only  a  limited  FTD  capability.  At  a  remote 
short  tour  base  such  as  Kunsan  AB,  if  a  maintenance  technician 
doesn't  receive  the  required  formal  FTD  training  prior  to 
arrival,  he/she  will  likely  depart  for  another  assignment  having 
never  received  the  training.  PACAFR  50-17  (1989)  directed  formal 
training  be  provided  to  Korean  units  via  Mobile  Training  Team 
(MTT)  or  temporary  duty  (TDY)  FTD  instructors,  as  well  as  via 
enroute  training,  although  travel  funds  were  frequently  not 
available  to  support  those  types  of  training.  Several 
interviewees  suggested  that  developing  the  capability  to  "bring 
the  classroom  to  the  students"  would  significantly  enhance 
unit-level  training. 

Aircraft  Maintenance  Qualification  Program  (AMQP).  AMQP  is 
a  centralized  three-phase  orientation  and  training  program 
designed  to  ensure  that  maintainers  arrive  at  their  unit  with  the 
skills  to  be  immediately  productive  and  provides  structured 
training  throughout  their  assignment  to  PACAF.  A  key  concept  in 
the  design  of  AMQP  is  centralization  of  training.  Recognizing 
that  mission  requirements  often  interfere  with  the  unit's  ability 
to  provide  effective  orientation  and  qualification  training,  AMQP 
attempts  to  conduct  training  in  an  environment  that  is  not  in 
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competition  with  sortie  production.  The  Training  Management 
Division  (TMD)  at  each  PACAF  base  is  the  primary  point-of -contact 
and  advisor  to  the  Deputy  Commander  for  Maintenance  ( DCM)  on  all 
maintenance  training  matters,  and  one  of  the  TMD  responsibilities 
is  to  manage  the  AMQP.  PACAFR  (Chapter  2)  50-17  describes  the 
organization,  responsibilities,  and  manpower  authorizations  of 
the  TMD. 

As  noted  earlier,  the  three-phase  AMQP  prepares  maintenance 
personnel  for  the  complex  and  critical  demands  of  providing 
mission  capable  and  safe  aircraft  for  the  training  of  combat 
crews.  Phase  I  must  be  completed  by  all  new  arrivals  and 
includes  unit  in-processing,  initial  interview/evaluation, 
general  maintenance  orientation  (i.e.,  block  training),  and 
ancillary  training.  The  initial  interview/evaluation  determines 
what  training,  if  any,  the  individual  requires  subsequent  to 
Phase  I.  Phase  II  is  formal  FTD  training  designed  to  provide 
practical  training  on  specific  systems.  This  training  serves  as 
a  foundation  for  entry  into  Phase  III  and  OJT.  Phase  III  is 
hands-on  task  oriented  training  (TOT)  on  the  actual  equipment. 
Completion  of  Phase  III  signifies  the  individual  is  prepared  to 
perform  specific  tasks  unsupervised  and  unassisted.  PACAF 
requires  completion  of  Phase  III  for  maintainers  in  AFSCs  452X4X 
and  462XX  assigned  to  the  flight-line.  Quality  Assurance  (QA) 
personnel  assigned  to  the  DCM  staff  ensure  the  quality  of 
training  by  conducting  task  evaluations  of  AMQP  Phases  II  and  III 
graduates  and  reporting  the  results  to  TMD. 


On-The-Job  Training  (OJT).  OJT  is  the  mainstay  of 
unit-level  training  throughout  the  Air  Force.  Enroute  training 
and  AMQP  do  not  replace  or  diminish  the  need  for  effective  OJT. 
Although  the  recent  revision  of  AFR  50-23  (1990)  has  changed  many 
of  the  policies  and  procedures  affecting  enlisted  specialty 
training  (EST),  the  concept  of  OJT  remains  virtually  unchanged. 
The  establishment  of  self-study  programs  such  as  career 
development  courses  (CDC)  as  a  separate  component  of  EST  is  the 
most  significant  change  affecting  OJT  (Note.  Figure  1  reflects 
this  recent  change  to  OJT). 

OJT  is  informal  training  conducted  at  an  individual's  duty 
location.  It  can  include:  (a)  hands-on,  over-the-shoulder 
instruction;  (b)  exportable  courseware  (e.g.,  interactive 
videodisc  [IVD]);  (c)  unstructured  self-training  (e.g., 
observation,  repeated  task  performance);  and  (d)  evaluation  and 
assessment  tools  used  to  identify  individual  training  needs 
(e.g.,  performance  evaluation  checklists,  written  knowledge 
tests).  Data  analyses  indicate  hands-on  training  is  the 
preferred  method  for  conducting  OJT.  Of  the  93  interviewees  who 
responded  to  the  question  concerning  the  use  of  different  types 
of  instructional  media /met hods,  all  (100%)  reported  the  use  of 
hands-on  instruction  at  the  unit  level  (Table  4).  Additionally, 
for  the  vast  majority  of  interviewees,  hands-on  was  also 
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considered  the  most  effective  method  of  instruction  (see  Table 
5). 

The  maintainers  often  expressed  a  need  for  ways  and  means  to 
facilitate  and  enhance  the  conduct  of  OJT,  and  specifically 
hands-on  skill  training.  When  asked  which  features  of  training 
technologies  were  most  desirable,  the  preference  was  for  those 
features  which  aided  in  the  training  of  difficult  tasks  such  as 
troubleshooting  and  allowed  for  hands-on  skill  development.  Most 
of  the  maintenance  technicians  recognized  that  hands-on  training 
and  OJT  were  the  core  of  the  training  program  at  the  unit  level. 
Although  respondents  were  often  vague  about  how  to  improve  OJT,  a 
few  suggestions  did  emerge,  such  as  more  standardization  of  task 
requirements  and  evaluations,  less  emphasis  on  recordkeeping, 
better  qualification  of  trainers,  stronger  leadership  support, 
and  better  feedback  to  training  planners,  managers,  and 
developers.  Most  of  the  suggestions  offered,  as  may  be  seen  in 
the  examples  just  given,  are  addressed  in  the  discussion  of  other 
training  need  or  problem  areas.  This  indicates  a  widely-held  and 
often  expressed  feeling  that  the  key  to  better  training  at  the 
unit  level  lies  in  the  OJT  program,  and  any  effort  designed  to 
increase  efficiency  in  unit-level  training  must  be  relevant  to 
the  OJT  process. 

Training  Resources:  Planning  and  Programming 

The  purpose  of  training  planning  and  programming  is  to 
determine  who,  what,  when,  and  where  to  train.  It  includes  a 
variety  of  macro- level  activities  which  must  occur  prior  to  the 
development  and  implementation  of  training.  One  of  the  first 
activities  in  planning  and  programming  is  the  analysis  of  jobs 
and  identification  of  those  job  tasks  which  require  training. 

Once  this  has  been  completed,  training  is  allocated  amongst  the 
various  training  settings  and  resources  are  programmed  to  support 
it. 

The  units  visited  during  this  project  were  not  involved,  to 
any  significant  degree,  with  the  training  planning  and 
programming  activities  noted  above.  These  functions  occurred  at 
higher  levels  within  the  AF,  i.e.,  the  MAJCOM  and  AF  level. 

Their  products,  such  as  Specialty  Training  Standards  (STS)  and 
Job  Qualification  Standards  (JQS),  were  filtered  down  to  the 
field  and,  in  turn,  used  by  units  to  manage  the  training 
requirements  of  their  personnel.  Consequently,  the  resources  for 
planning  and  programming  at  the  unit  level  were  very  limited. 

Problems  did  exist,  however,  with  the  planning  and 
programming  products  sent  down  to  the  units.  In  particular, 
there  appeared  to  be  a  lack  of  symmetry  between  the  perceived 
training  requirements  and  those  specified  in  planning  documents. 
Respondents  identified  a  significant  amount  of  training  as  being 
conducted  to  an  inappropriate  level  (i.e.,  not  needed, 
overtrained,  undertrained,  or  needed  but  not  available).  A 
number  of  maintainers,  for  example,  felt  that  some  training 
requirements,  such  as  Hangar/Arch  Door  training.  Egress  training. 
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and  Block  training,  were  either  not  needed  or  overtrained.  These 
sentiments  pertained  to  ancillary  training  requirements  in 
particular  (which  are  predominantly  taught  at  MAT),  although 
examples  of  unnecessary  job-related  training  were  also  cited. 
Respondents  felt  that  some  training  requirements  were  unnecessary 
because  they  simply  were  not  relevant  to  their  AFSs.  Some 
respondents  also  felt  that  they  performed  tasks  frequently  enough 
on  the  job  and  did  not  need  further  training. 

The  maintenance  technicians  also  noted  an  absence  or  lack  of 
training  in  some  areas.  FTD  Qualification  and  CAMS  training  were 
the  areas  most  frequently  cited  as  not  being  trained.  According 
to  the  technicians,  the  courses  were  either  not  offered  or  a 
qualified  instructor  was  not  available  to  teach  them.  Areas  that 
were  considered  undertrained  included  Self -Aid  and  Buddy  Care, 
Chemical  Warfare,  CPR,  CAMS,  and  troubleshooting  skills  training. 
Maintainers  saw  these  areas  as  critical  to  their  jobs  and,  thus, 
deserving  of  more  attention.  This  training,  according  to  those 
interviewed,  should  ideally  occur  prior  to  arrival  at  a  duty 
station  (i.e.,  enroute).  Tables  D-4  to  D-7  list  the  types  of 
training  identified  as  being  conducted  inappropriately. 

The  apparent  asymmetry  between  actual  and  perceived  training 
requirements  concerned  OJT  trainers  and  workcenter  supervisors. 
There  was  a  perception  that  training  needs  assessment  has  not 
kept  pace  with  changes  in  job  requirements.  The  DCM  at  one  of 
the  data  collection  sites  communicated  a  need  to  be  able  to 
accurately  determine  exactly  what  skills  and  knowledges  are 
required  of  each  job  in  the  maintenance  complex  so  that  each 
incumbent  could  receive  the  appropriate  training  for  the  job. 

Any  unnecessary  training  was  counterproductive  and  debilitating, 
especially  with  the  current  workload  and  manning  constraints. 
Nonetheless,  these  supervisors  did  not  feel  comfortable  in  making 
training  content  decisions  that  might  impact  a  trainee's  career 
progression.  They  were  either  not  aware  of  how  these  decisions 
were  made  at  the  macro  levels  (i.e..  Utilization  and  Training 
Workshop,  OSR,  Ancillary  Training  Review  Panel)  or  felt  those 
procedures  were  to  complex  to  use  at  the  workcenter  level. 

Training  Resources:  Development 

Training  development  activities  have  typically  been  the 
responsibility  of  organizations  such  as  ATC  or  other  MAJCOMs. 
Training  is  developed  using  products  from  the  planning  and 
programming  phase,  such  as  job  analyses,  core  task  lists,  and 
task  training  requirements.  Development  includes  activities  such 
as  test  development,  selection  of  instructional  media  and 
methods,  and  development  and  validation  of  instructional 
materials . 

The  resources  available  for  training  development  at  the 
units  in  this  study  were  minimal.  The  primary  development  tool 
noted  by  interviewees  was  the  ISD  model  as  presented  in  AFM  50-2 
and  AFP  50-58.  However,  these  documents  were  difficult  for 
laymen  to  understand  and  apply,  and  the  availability  of  personnel 
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with  ISD  expertise  rapidly  declined  as  one  moved  from  FTD  to  MAT 
to  the  maintenance  unit  itself.  In  fact,  very  few  personnel  at 
the  maintenance  squadrons  even  knew  what  ISD  was. 

The  availability  of  training  development  expertise  at  each 
location  directly  impacted  the  amount  of  training  development 
actually  done.  This,  of  course,  limited  the  ability  of 
instructors  to  tailor  their  own  training  programs.  Yet,  despite 
all  of  this,  the  maintenance  instructors  expressed  little  concern 
over  the  lack  of  resources  for  training  development.  They  were 
either  satisfied  with  the  courses  they  had  or  developed 
themselves,  or  they  felt  that  development  was  the  responsibility 
of  others,  such  as  ATC. 

Training  Resources ;  Delivery 

The  description  of  training  capability  in  AFR  50-23  (1990) 
includes  availability  of  resources  as  a  primary  consideration 
when  determining  a  unit's  capability  to  provide  training.  The 
key  resources  to  consider  are  equipment,  qualified  instructors, 
facilities,  and  training  materials  such  as  study  references 
audio-visual  materials . 

Equipment .  A  key  responsibility  of  the  TMD  is  to  ensure 
theavailability  of  equipment  resources  to  support  TMD  training 
(e.g.,AMQP  Phases  I  and  III),  FTD  training  (e.g.,  enroute 
training,  AMQP  Phase  II),  and  mobile  training  teams  (MTT) .  TMD 
identifies  requirements  and  schedules  equipment,  including 
aircraft,  in  the  Monthly  Maintenance  Plan.  Interviewee  comments 
indicated  equipment  availability  was  not  a  significant  issue  for 
SLT. 

Qualified  Instructors/Unit-Level  Trainers.  Instructors  and 
unit-level  trainers  who  are  subject  matter  competent  and 
proficient  in  conducting  training  are  critical  to  an  effective 
training  program.  Policies  and  procedures  are  established  by  AFR 
50-54  (1987),  PACAFR  66-5  (1983),  and  PACAFR  50-17  (1989)  to 
ensure  availability  of  qualified  instructors  for  FTD  and  TMD 
training  in  PACAF.  Maintenance  instructors  (MI)  are  selected 
based  upon  their  maintenance  experience  and  skill  as  well  as 
their  motivation  to  teach.  Both  FTD  and  TMD  are  authorized 
permanent,  full-time  instructor  staffs.  The  number  of  authorized 
positions  is  relative  to  maintenance  AFSC  populations  at  the 
base.  FTDs  and  TMDs  may  also  assign  temporary  Mis  when  training 
requirements  warrant.  In  addition,  each  unit  assigns  its  own 
personnel  to  act  as  OJT  trainers  for  less  experienced 
maintainers.  The  following  discussion  focuses  on  the  primary 
instructor-  and  trainer-related  problems  found  at  the  unit  level. 

A  major  problem  at  maintenance  units  was  the  unavailability 
of  OJT  trainers  brought  about  by  competing  commitments. 
Deployments  and  exercises,  in  particular,  often  reduced  the 
number  trainers  at  the  unit  level.  The  importance  of  the 
operational  mission  during  deployments  and  evaluations  (or 
competition)  during  exercises  necessitated,  for  the  most  part, 
that  only  the  most  qualified  personnel  attend.  Consequently, 
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trainers  are  removed  from  the  squadron  while  the  less  experienced 
maintainers  are  left  behind.  At  Kunsan  AB,  this  problem  was 
compounded  because  of  the  absence  of  an  FTD.  The  FTD's  training 
burden  has  shifted  to  the  unit  OJT  programs,  many  of  which  are 
already  burdened  as  a  result  of  the  lack  of  experience  of 
incoming  personnel. 

FTDs  also  suffered  from  limited  availability  of  instructors 
and  could  not  meet  the  needs  of  the  field.  The  demands  of  the 
operational  units  simply  exceeded  the  number  of  available 
instructors,  and  therefore,  many  of  the  trainees  were  not 
receiving  the  training  they  required.  When  asked  what  limited 
FTD  training,  56%  of  the  respondents  cited  a  lack  of  qualified 
instructors . 

Another  problem  concerned  the  technical  qualifications  of 
OJT  trainers.  Some  trainers,  while  available  at  the  unit,  lacked 
the  technical  expertise  necessary  to  train  certain  tasks.  Rivet 
Work  Force  has  contributed  to  this  problem  by  merging  two  or  more 
AFSs  and  making  trainers  at  the  unit  level,  in  particular, 
responsible  for  knowing  and  instructing  tasks  for  which  they 
themselves  have  not  received  sufficient  training.  As  a  result, 
many  of  the  trainers  are  learning  tasks  at  the  same  time  they  are 
attempting  to  instruct  others  on  them. 

Personnel  turnover  and  shift  work  accounted  for  another  OJT 
trainer-related  problem.  Because  of  short-tour  lengths  and  swing 
shifts,  trainees  often  had  several  different  trainers.  As  a 
result,  trainers  had  very  little  knowledge  of  their  trainee's 
capabilities,  experience,  strengths  and  weaknesses.  The  trainers 
also  had  different  styles  of  instruction,  some  good  and  some  not 
so  good.  This  situation  created  considerable  discontinuity  and 
impeded  the  training  process. 

Finally,  quite  a  few  OJT  trainers  were  not  trained  to  be 
instructors  and  lacked  necessary  teaching  skills.  While  these 
instructors  are  technically  competent,  they  are  not  very 
effective  instructors.  Less  than  a  quarter  of  the  OJT  trainers 
interviewed  received  any  formal  course  in  technical  instruction. 

A  majority  of  the  instructors  and  trainers  in  our  study  received 
no  formal  training  at  all  or  attended  the  OJT  Trainer/Supervisor 
Course  which,  according  to  the  respondents,  did  not  adequately 
teach  them  instructional  techniques. 

Several  OJT  trainers  raised  an  issue  concerning  their 
qualifications  to  train.  They  stated  that,  although  they  felt 
subject  matter  qualified,  they  received  no  or  inadequate  training 
on  how  to  conduct  OJT  (Appendix  D,  Table  D-9).  When  asked  how 
they  were  trained  or  qualified  to  instruct,  8  of  22  OJT  trainers 
(36%)  responded  that  they  received  no  formal  training  to  be  an 
OJT  trainer.  Several  others  who  attended  the  OJT 
Trainer/Supervisor  course  at  FTD  felt  the  course  did  not 
adequately  prepare  them  because  it  emphasized  documentation  of 
OJT  rather  than  instruction. 

Facilities.  Training  facilities  did  not  appear  to  be  an 
issue  at  any  of  the  data  collection  bases.  Both  FTD  and  TMD  have 
dedicated  office  and  classroom/laboratory  space  at  each  of  the 
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data  collection  sites.  Most  of  the  squadrons  have  no  dedicated 
classroom/ laboratory  space  and  must  use  unit  conference  rooms  if 
training  requires  a  classroom- type  setting.  Since  most  of  the 
unit-level  training  at  the  work  center  is  OJT,  dedicated  training 
facilities  are  neither  required  nor  practical. 

Training  Materials.  Availability  of  training  materials  was 
not  an  issue  for  most  interviewees.  FTD  and  TMD  staffs  developed 
training  materials  to  support  their  courses  and  there  was  heavy 
reliance  on  technical  data  (e.g.,  technical  orders  [TO])  in  all 
maintenance  training.  The  primary  concern  among  interviewees  was 
that  some  training  materials  were  out-of-date.  Most  complaints 
centered  on  videotapes  used  at  TMD.  Many  of  the  tapes  have  been 
used  to  support  training  for  years  and  they  depict  obsolete 
equipment  and  procedures.  Interviewees  questioned  the 
credibility  and  quality  of  training  provided  by  such  materials. 
Finally,  there  was  some  concern  over  the  lack  of  Career 
Development  Courses  (CDCs)  for  rivetized  specialties. 

Media/Methods .  OJT  tended  to  rely  heavily  on  self-paced 
hands-on  instruction  supported  by  text  (e.g.,  TOs,  CDCs).  FTD 
and  TMD  courses  were  typically  conducted  in  classroom/ laboratory 
settings,  and  were  supported  by  a  variety  of  dedicated  training 
media,  and  tended  to  be  group-lockstep  or  group-paced.  Dedicated 
training  media  were  generally  not  available  in  the  units.  The 
types  of  media/methods  available  to  conduct  SLT  and  the  percent 
of  respondents  using  them  are  contained  in  Table  4;  ratings  of 
the  effectiveness  of  instructional  media  and  methods  are 
displayed  in  Table  5. 

The  only  issue  identified  that  involved  media  or  methods 
was  the  lack  of  availability  of  IVD.  Each  of  the  bases  visited 
had  IVD  courseware,  but  only  one  base  appeared  to  be  using  it  for 
SLT  and  that  base  was  using  it  minimally.  Although  IVD 
application  is  limited  because  the  bases  had  only  a  few  tasks 
available  for  training,  several  TMD  staff  members  interviewed  had 
no  guidance  or  training  on  how  to  integrate  IVD  into  their 
training  when  rrurseware  becomes  available.  Additionally,  most 
of  the  unit-level  interviewees  (i.e.,  OJT  trainers,  training 
monitors,  shop  chiefs,  training  coordinators)  were  either  unaware 
that  IVD  courseware  was  available  on  base  or  that  the  technology 
even  existed. 

This  availability  issue  also  reflects  a  potential  user 
acceptability  concern  which  may  require  further  study.  The 
problem  of  successfully  transporting  IVD  to  the  unit  highlights 
the  need  for  training  technology  developers  and  programs  managers 
to  plan  integration  of  new  technologies  into  the  user  community 
and  provide  mechanisms  for  support.  If  trainers  are  reluctant, 
for  whatever  reason,  to  use  an  available  technology  such  as  IVD, 
they  are  likely  to  be  just  as  reluctant  to  use  other 
technologies.  Table  D-10  in  Appendix  D  presents  specific 
technologies  and  features  that  are  desired  by  unit-level  training 
personnel . 


Training  Resources :  Management 


Management,  as  it  is  used  here,  refers  to  the  day-to-day 
management  of  training  operations.  Management  activities  include 
managing  individual  training  requirements,  scheduling  students 
and  instructional  resources,  and  tracking  student  progress. 
Training  management  responsibilities  are  shared  by  the  TMD  and 
training  monitors  at  each  of  the  maintenance  units.  The  units  in 
this  study  had  several  management  information  sources  and  tools 
at  their  disposal  for  accomplishing  these  activities. 

Individual  Training  Reguirements .  Management  of  individual 
training  requirements  involves  the  initial  identification  of 
training  requirements  and  the  management  and  tracking  of  those 
requirements  thereafter.  For  the  initial  identification  of 
individual  training  requirements,  units  relied  predominantly  on 
individual  interviews,  personnel  training  records  (AF  Form  623 
and  797),  and  CAMS  reports.  The  personnel  training  records  and 
CAMS  reports  also  served  as  the  primary  tools  for  managing  and 
tracking  training  requirements. 

Respondents  reported  no  significant  problems  with  the  tools 
and  procedures  for  tracking  training  requirements.  They  also 
felt  comfortable  with  the  initial  evaluations  of  training 
requirements  for  incoming  individuals.  However,  as  noted  above, 
training  managers  were  uncomfortable  making  decisions  about 
training  requirements  for  individuals  which  deviated  from  those 
listed  in  the  planning  and  programming  documents,  e.g.,  STS  or  AF 
Form  623. 

Scheduling  Students  and  Resources.  The  resources  for 
scheduling  both  students  and  instructional  resources  were 
limited.  The  scheduling  of  students  was  accomplished  using  CAMS 
as  well  as  correspondence  between  the  TMD  and  individual  units. 
For  example,  units  had  to  submit  requests  for  formal  courses 
(e.g.,  FTD  courses)  to  the  TMD  who  then  forwarded  the  request  to 
ATC  for  coordination,  and  vice-versa.  In  the  end,  however, 
scheduling  was  reduced  to  last  minute  phone  calls  between 
schedulers  and  units. 

Equipment  scheduling  was  done  predominantly  through  the 
Monthly  Maintenance  Plan.  There  did  not  seem  to  be  a  problem 
with  this  function. 

Training  Resources:  Evaluation 

Evaluations  are  critical  to  the  success  of  an  instructional 
program.  They  provide  feedback  to  instructional  designers  about 
he  effectiveness  of  their  courses.  There  are  two  general  types 
of  evaluations,  internal  and  external.  Internal  evaluations  are 
based  on  student  performance.  They  reveal  the  effectiveness  of 
instruction  within  the  instructional  environment.  External 
evaluations,  on  the  other  hand,  are  based  on  job  performance  and 
are  an  indication  of  training  transfer.  These  evaluations  also 
provide  valuable  feedback  to  individuals  or  units.  They  can,  for 
example,  be  used  to  determine  whether  or  not  an  individual  has 


24 


successfully  acquired  the  requisite  knowledges  and  skills  in  a 
training  course,  or  they  can  be  used  to  measure  unit  readiness. 

Based  on  the  data  collected  in  this  study,  maintenance 
units  had  very  few  resources  for  evaluating  either  the  internal 
(i.e.,  student  performance)  or  external  validity  (i.e.,  job 
performance)  of  their  training.  Internal  validity  was 
predominantly  evaluated  using  paper  and  pencil  tests  (for 
ancillary  training),  course  critiques,  and  over-the-shoulder 
evaluations  based  on  Technical  Orders  (for  job-related  training). 
The  over-the-shoulder  evaluations  were  loosely  structured  and 
often  varied  from  individual  to  individual.  External  validity 
was  evaluated  using  follow-up  surveys,  e.g..  Training  Evaluation 
Reports  (TER),  Training  Quality  Reports  ( TQR ) ,  and  local 
questionnaires,  and  informal  communications  with  units.  TMDs, 
FTDs,  and  Tech  Schools  would  typically  send  out  surveys  to 
graduates  and  their  supervisors  60  to  90  days  after  graduation. 
These  surveys  solicited  feedback  on  the  appropriateness  and 
effectiveness  of  the  course.  Additionally,  the  maintenance 
community  used  QA  personnel  to  evaluate  unit  readiness,  as  well 
as  AMQP  graduates.  These  evaluations  provided  valuable  job 
performance  feedback  to  the  AMQP  courses,  in  particular,  and  the 
unit-level  training  system,  in  general. 

The  interviewees  did  not  express  any  significant  concerns 
over  the  effectiveness  of  training  evaluation  procedures  with 
regard  to  student  performance.  They  did,  however,  express  some 
concern  over  the  effectiveness  of  procedures  for  evaluating  the 
external  validity  of  training.  For  example,  several  maintainers 
had  doubts  about  the  effectiveness  of  the  follow-up  surveys. 

They  felt  that  these  surveys  failed  to  communicate  training 
requirements  back  to  the  training  community. 

Many  of  the  interviewees  also  felt  that  the  Quality 
Assurance  (QA)  evaluations  were  both  not  standardized  and  poorly 
planned,  and  consequently,  needed  improvement.  According  to 
these  individuals,  the  evaluations  were  not  consistent  across 
evaluators.  Different  raters  would  evaluate  the  same  task  using 
different  criteria.  Also,  these  evaluations  were  seemingly 
performed  on  the  same  sets  of  individuals  each  time.  Personnel 
on  swing  shifts,  for  example,  were  rarely  evaluated.  Finally, 
many  of  the  interviewees  (including  QA  personnel  themselves)  felt 
that  QA  personnel  often  lacked  the  technical  competence  needed  to 
evaluate  many  of  the  job  tasks.  This  problem  was  largely  due  to 
the  additional  requirements  imposed  on  QA  personnel  by  Rivet 
Workforce.  Many  of  them  had  only  recently  learned,  or  were  in 
the  process  of  learning,  new  sets  of  tasks  which  they  were 
required  to  evaluate. 
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Table  4 


Frequency  of  Use  of  Instructional  Media/Methods 


%  of  Respondents 

Instructional  Using  Media/Method 

Media /Method  For  Unit  Level  Training 


Lecture/Presentation 

84.2 

Overhead 

67.4 

CAI 

10.5 

Sound-on-Slide 

56.8 

Video  Tape 

78.9 

IVD 

12.6 

Text 

97.9 

Hands-On 

100.0 

Mockup 

5.3 

Table  5 

Ratings  on  the  Effectiveness  of  Instructional  Media/Methods 


Mean 

Instructional  Effectiveness 

Media/Method  ( 1-7  Scale ) 


Mean 

SD 

N 

Lecture /Presentation 

5.2 

1.4 

83 

Overhead 

4.4 

1.4 

71 

CAI 

4.5 

2.1 

15 

Sound-on-Slide 

4.3 

1.5 

57 

Video  Tape 

5.3 

1.3 

77 

IVD 

5.2 

1.6 

14 

Text 

5.7 

1.3 

91 

Hands-On 

6.7 

.7 

94 

Mockup 

7.0 

0.0 

4 

Note.  Scale  values  for  effectiveness  ranged  from  1  (Very 
Ineffective)  to  7  (Very  Effective). 
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C.  Factors  Impeding  or  Facilitating  Training 

This  section  focuses  on  identification  of  factors  which 
either  limit  or  enhance  unit  level  training  beyond  those  factors 
directly  related  to  training  requirements  or  training 
capabilities.  The  ability  to  train  effectively  and  efficiently 
is  determined  by  a  wide  variety  of  external  factors  such  as  the 
time  available  to  conduct  training,  competition  for  available 
resources,  organizational  structure,  and  so  forth.  Altering  any 
of  these  factors  can  conceivably  change  the  effectiveness  and 
efficiency  of  training.  One  goal  of  the  SLT  Research  Initiative 
is  to  identify  those  HSD  technologies  which  best  enhance 
facilitating  factors  or  reduce  impeding  factors. 

Impeding  Factors 

Availability  of  Time.  Lack  of  adequate  time  to  train  was  a 
major  issue  for  interviewees.  When  asked  if  there  was  adequate 
time  available  for  training,  42%  (47  of  111)  responded  that  there 
was  not  sufficient  time  available  to  train  at  the  unit.  Only  12% 

(9  of  74)  and  3%  (3  of  105)  stated  there  was  not  sufficient  time 
available  to  train  at  FTD  and  TMD  respectively.  Of  the  47 
interviewees  who  denied  having  enough  time  to  train  at  the  unit, 

40  (85%)  identified  mission  requirements /operational  commitments. 
Lack  of  emphasis  on  training  (10  of  47,  21%),  non-availability  of 
trainers /trainees  due  to  TDY,  leave,  details,  and  so  on  (8  of  47, 
17%),  and  decreased  manning  levels  (7  of  47,  15%)  were  also 
identified  as  time  limiting  factors  (Appendix  D,  Table  D-ll). 

The  great  difference  in  training  time  available  at  the  unit 
versus  both  FTD  and  TMD  was  due  to  the  emphasis  placed  on  meeting 
FTD  and  TMD  training  requirements  and  mission  requirements. 

PACAFR  50-17  (1989)  assigns  the  TMD  responsibility  for  scheduling 
unit  personnel  for  FTD  and  TMD  training.  The  regulation  directs 
unit  commanders  to  ensure  their  personnel  attend  training.  The 
regulation  also  prescribes  the  policy  that,  while  unit  personnel 
are  in  scheduled  training,  "FTD  and  AMQP  training  must  be  given 
priority  and  will  not  be  canceled  when  local  exercises  are 
conducted"  (p.  4).  As  a  result  of  this  emphasis  on  completion  of 
training  outside  of  the  unit,  supervisors  and  trainers  "make 
time"  for  FTD  and  TMD  training,  but,  mission  requirements 
generally  impeded  their  ability  to  devote  enough  time  to  training 
at  the  unit. 

OJT  tended  to  suffer  the  most  under  these  conditions. 

Several  interviewees  admitted  that,  due  to  the  flying  schedules, 
there  was  little  time  for  unit-level  training  in  those  AFSCs 
which  directly  supported  launch  and  recovery  of  aircraft.  One 
senior  officer  expressed  deep  concern  for  the  lack  of  opportunity 
to  conduct  maintenance  training  and  stated  a  strong  case  for  ”... 
either  biting  the  bullet  and  flying  less  or  scheduling  the  flying 
differently."  This  sentiment  was  echoed  by  other  officers  and 
NCOS  at  various  levels  within  the  maintenance  management  community. 
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Manning  and  Personnel  Policies.  Manning  and  personnel 
policies  combine  to  impede  the  ability  to  provide  adequate 
training,  particularly  at  short  tour  bases.  All  of  the  PACAF 
bases  visited,  including  HQ  PACAF,  were  reducing  manning  levels 
to  85%  of  authorized  strength.  Some  units  reported  manning 
levels  below  85%  during  the  data  collection  period.  There  also 
appeared  to  be  an  increase  in  the  number  of  new  arrivals  with 
little  or  no  experience  with  F-16  maintenance.  The  resulting 
situation  was  one  in  which  training  requirements  were  increasing 
as  the  ability  to  conduct  traditional  training  is  decreasing. 

Most  of  the  interviewees  at  each  base  recognized  this  as  a 
significant  issue.  When  asked  for  proposed  alternatives  to  this 
problem,  69%  of  those  responding  (24  of  35)  recommended  that  the 
personnel  assignment  policies  should  be  changed  to  reduce  the 
number  of  new  arrivals  who  lack  experience  in  the  weapons  system 
(e.g.,  technical  school  graduates,  cross-command/weapon  system 
re-assignments)  (Appendix  D,  Table  D-2).  Specifically,  they 
recommended  assigning  maintainers  to  short  tour  bases  according 
to  special  experience  identifiers  (SEI)  as  well  as  skill  level  so 
that  only  personnel  with  appropriate  weapon  system  experience 
would  be  assigned,  reducing  the  need  for  basic  systems  training. 
They  were  particularly  adamant  about  not  assigning  3-levels  to 
short  tour  areas  because  they  are  minimally  productive  and  do  not 
receive  adequate  training.  As  one  NCO  stated,  "By  the  time  we 
get  them  trained,  if  we  get  them  trained  at  all,  they  are  ready 
to  leave"  (refer  to  Table  3).  There  is  also  genuine  concern  on 
the  part  of  supervisors,  trainers,  and  trainees  that  the  training 
most  3-levels  receive  at  short  tour  bases  puts  them  at  a  distinct 
disadvantage  with  regard  to  career  progression  when  compared  to 
their  contemporaries  assigned  to  long  tour  overseas  or 
continental  U.S.  (CONUS)  bases.  Specific  effects  of  personnel 
turnover  cited  by  participants  are  listed  in  Table  D-12  (Appendix 
D). 

Lack  of  Funding.  An  obvious  constraint  in  today's 
environment  is  the  budget.  Although  budget  cutbacks  can  be 
associated  with  a  number  of  training  problems,  one  problem  in 
particular  stands  out  in  this  analysis  —  the  lack  of  TDY  funds. 
This  situation  has  greatly  restricted  access  to  qualified 
instructors.  Kunsan  AB,  as  noted  earlier,  has  no  FTD  and, 
therefore,  FTD  instructors  from  other  bases  must  travel  to  Kunsan 
AB  to  conduct  training.  Unfortunately,  interviewees  contended 
that  the  travel  funds  were  not  readily  available  to  bring 
instructors  from  other  bases  nor  send  individuals  from  Kunsan  AB 
TDY  to  be  trained  at  an  FTD.  As  a  result,  much  of  the 
qualification  and  continuation  training  was  not  accomplished  or 
the  quality  of  training  suffers.  The  non-availability  of 
qualified  instructors  at  Kunsan  AB  illustrated  the  importance  of 
enroute  training. 
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Facilitating  Factors 


Attitude  and  Motivation.  Not  enough  can  be  said  about  the 
positive  attitude  and  motivation  of  the  aircraft  maintenance 
community.  They  have  been  "doing  more  with  less"  for  so  long 
that  it  has  become  almost  routine  for  them  to  overcome  adversity. 
This  was  particularly  true  at  the  SLT  data  collection  sites. 
Regardless  of  the  impediments  to  conducting  training,  supervisors 
and  trainers  seemed  to  always  find  a  way  to  both  get  the  job  done 
and  to  train  their  people.  Motivated  trainers  and  students 
willing  to  learn  keep  the  training  programs  moving  forward. 

Changes  to  OJT.  As  noted  earlier,  AFR  50-23  (1990)  was 
recently  revised  resulting  in  substantial  changes  to  EST  which 
should  facilitate  training  signiricantly .  Some  of  the  structure 
has  been  removed  from  the  program  and  some  regulatory 
requirements  have  been  changed  which  should  permit  more 
flexibility  in  planning,  developing,  conducting,  managing,  and 
evaluating  unit-level  training.  Many  of  the  details  of  EST  were 
purposely  left  to  the  people  in  the  training  system  who  best 
understand  unit-level  training  and  can  best  determine  specific 
training  needs.  Perhaps  the  most  beneficial  change  in  EST  was  to 
put  more  emphasis  on  evaluation  of  job  performance  rather  than 
paperwork  to  determine  effectiveness  of  training.  In  addition, 
added  emphasis  has  been  placed  on  preparing  OJT  trainers  to  plan, 
develop,  deliver,  and  evaluate  training. 

Increased  Commitment  to  Maintenance  Training.  There 
appears  to  be  a  growing  commitment  at  various  levels  within  the 
Air  Force  to  providing  the  best  training  possible,  particularly 
for  the  maintenance  community.  This  research  initiative  is 
partial  evidence  of  that  commitment.  Faced  with  reductions  in 
manning  and  budget  cuts,  senior  officials  are  relying  on  enhanced 
training  to  ensure  that  mission  requirements  are  met. 
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D.  Summary  of  Major  Training  Problems 

Table  6  summarizes  the  apparent  deficits  or  problems  in 
training  for  PACAF  F-16  maintenance  units  as  seen  in  the  analysis 
of  unit  level  training  requirements,  training  capabilities,  and 
those  factors  influencing  SLT.  Each  of  the  issues  listed  in 
Table  6  is  discussed  more  fully  in  the  preceding  sections  of  this 
chapter.  These  deficits  are  indicative  of  specific  areas  within 
the  training  system  which  may  potentially  benefit  from  technology 
applications.  For  example,  the  problem  of  Inappropriate  Training 
Requirements  may  indicate  a  poor  training  requirements  selection 
process  and/or  poor  management  of  individual  training 
requirements.  These  issues  will  l>e  discussed  more  fully  in  the 
next  chapter. 


Table  6. 

Summary  of  Major  Training  Problems  For  PACAF  F-16  Maintenance 
Squadrons 


1.  Inappropriate  Training  Requirements  (e.g..  Lack  of 
Troubleshooting,  Basic  Systems  Knowledge,  and 
Computer  Training) 

2.  Lack  Of  Qualified  Instructors /OJT  Trainers 

3.  Manpower  and  Personnel  Policies  (e.g.,  Rivet  Workforce, 
Cross-Command/Weapon  System  Assignments,  Manning) 

4.  Need  For  Enhanced  Hands-On  Training/OJT 

5.  Lack  of  Training  Opportunities  Due  To  Improved  Weapon 
System  Technology 

6.  Limited  Training  Time  Due  To  Operational  Commitments 

7.  Lack  of  Enroute  Training 

8.  Inadequate  Feedback  Mechanisms  For  Training  Courses 

9.  Inadequate  Job  Performance  Measurement  Procedures 

10.  Lack  of  TDY  Funding 

11.  Implementation  Problems  With  New  Training  Media 
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IV.  RELATING  TRAINING  SYSTEM  NEEDS 
TO  TRAINING  TECHNOLOGIES 

This  chapter  describes  the  process  by  which  the  needs  of  the 
unit-level  training  system  were  systematically  identified, 
documented  and  matched  to  HSD  training  technologies.  In  the 
previous  chapter,  researchers  identified  a  number  of  general 
problems  in  the  unit-level  training  system  for  PACAF  F-16 
maintenance  units.  Researchers  described  these  problems  with 
broad  narrative  statements  based  on  the  results  of  their  content 
analyses.  However,  these  broadly-defined  problems,  while 
insightful  in  and  of  themselves,  required  further  analysis  before 
they  could  be  related  to  training  technologies.  This  chapter 
documents  the  efforts  of  researchers  to  identify  the  specific 
training  system  functions  implicated  by  these  problem 
descriptions  as  being  deficient,  and  subsequently,  to  relate  them 
to  the  functions  of  HSD  training  technologies. 

The  impetus  for  this  line  of  Air  Force  research 
lies  in  the  certainty  that  force  structure  and  budgetary  levels 
will  continue  the  trend  of  systematic  reductions  and  short-term 
adjustments  to  immediate  fiscal  shortfalls.  This  activity 
constitutes  an  intermediate  step  in  the  development  of  a  long- 
range  S&T  Investment  Strategy  with  the  establishment  of  a 
framework  with  which  to  focus  the  emerging  technologies  against 
concrete  user  needs.  Matching  of  training  needs  and  technologies 
is  only  a  preliminary  step,  however,  since  this  phase  of  the 
research  initiative  examined  only  a  narrow  segment  of  the  Air 
Force  training  domain  (i.e.,  F-16  aircraft  maintenance  units 
deployed  in  the  Pacific  theater).  Accordingly,  suggestions  will 
be  made  for  further  research  that  will  attempt  to  generalize  the 
results  of  this  data  collection  across  a  larger  segment  of  the 
Air  Force's  vast  training  system. 

A.  Analytic  Frameworks  ISD  Model 

A  review  of  available  training  literature  and  AF  training 
research  revealed  that  no  clear-cut  or  systematic  methodologies 
existed  for  identifying,  documenting  and  relating  training  system 
needs  to  training  technologies  (Carson,  Chambers  &  Gosc,  1984; 
Chenzoff  et  al.,  1984;  Stephenson  &  Burkett,  1975).  Most  of  the 
research  in  this  area  has  focused  on  training  needs  assessment  and 
has  not  attempted  to  identify  and  relate  the  needs  of  the  training 
system  to  potential  technology  solutions. 

This  study  adopted  a  "training  system  needs  assessment” 
approach  as  opposed  to  a  traditional  training  needs  assessment 
approach.  The  distinction  between  the  two  is  important  here.  On 
the  one  hand,  a  training  needs  assessment  "provides  information  on 
where  training  is  needed,  what  the  content  of  the  training  should 
be,  and  who  within  the  organization  needs  training  in  certain  kinds 
of  skills  and  knowledge  (Ostroff  &  Ford,  1989).”  This  type  of 
assessment  stresses  the  importance  of  three  interrelated 
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components:  organization,  task,  and  person.  A  training  system 
needs  assessment,  on  the  other  hand,  provides  information  on  which 
processes  or  functions  in  the  training  system  need  improvement. 
These  processes  can  be  broadly  stated  as  planning,  programming, 
management,  evaluation,  development  and  delivery  of  training. 
Consequently,  training  needs  assessment  would  be  one  process  within 
this  system. 

The  basic  analytic  framework  chosen  for  this  research  was  the 
Instructional  System  Development  (ISD)  model.  This  model  had 
several  features  which  made  it  appropriate  for  this  purpose. 
First,  the  ISD  model  covered  a  wide  range  of  training  activities, 
including  planning,  programming ,  development,  delivery, 
management,  and  evaluation.  Moreover,  the  model  can  be  defined 
at  such  a  level  as  to  differentiate  between  the  unique  functions 
of  various  training  technologies.  Many  technologies,  while 
somewhat  similar,  are  intended  to  enhance  different  aspects  of 
the  same  general  training  area.  For  example,  two  technologies 
might  enhance  training  management  but  do  so  in  qualitatively 
different  areas,  such  as  resource  scheduling  and  student 
tracking.  A  broadly  defined  framework  would  not  capture  these 
nuances.  Fortunately,  the  ISD  model  allowed  us  to  describe 
training  technologies  and  research  at  a  meaningful  and 
analytically  useful  level. 

The  ISD  model  is  also  widely  accepted  within  the  AF  and 
other  training  communities,  although  the  exact  form  of  this  model 
is  a  matter  of  continuing  debate.  AFM  50-2  and  AFP  50-58,  in 
particular,  formed  the  basis  of  the  framework.  The  final 
version,  as  seen  in  figure  3,  was  the  result  of  repeated  reviews 
and  modifications  by  AL/HRT  ISD  experts.  Each  of  the  processes 
is  defined  in  Appendix  F. 

B.  Applying  the  Analytic  Framework 

The  process  of  matching  training  technologies  to  training 
system  needs  occured  in  three  steps.  In  the  first  step, 
researchers  developed  profiles  for  a  selected  group  of  training 
technologies  based  on  the  analytic  framework  shown  in  figure  3. 
The  "technology  profiles"  contained  the  specific  ISD  processes 
(i.e.,  training  functions)  which  were  enhanced  by  a  technology. 
Next,  researcher 8  examined  the  general  training  system  problems 
identified  during  analysis  of  the  interview  data  to  determine  the 
specific  ISD  processes  which  needed  improvement.  As  such,  the 
"training  system  problem  profiles"  contained  the  specific  ISD 
processes  which  needed  improvement  according  to  the  researcher's 
interpretation  of  the  general  problem  description.  In  the  final 
step,  the  technology  profiles  were  compared  to  the  training  system 
problem  profiles,  and  areas  of  overlap  and  non-overlap  were  noted. 
Following  is  a  discussion  of  the  procedures  and  results  of  this 
matching  process. 
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Figure  3.  Analytic  Framework  Based  On  Instructional  System 
Development  Model 
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(Figure  3  continued) 
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Training  Technology  Profiles 


HSD  has  a  number  of  training  technologies  currently  under 
development  (Appendix  E).  The  impetus  for  training  technology 
research  comes  from  a  number  of  different  sources  including 
formal  requests  from  AF  organizations  as  well  as  self-initiated 
research  in  response  to  anticipated  needs.  Yet,  whatever  their 
source  is,  these  technologies  have  one  goal  in  common,  to  improve 
the  effectiveness  or  efficiency  of  one  or  more  parts  of  the 
training  system.  The  objective  of  this  step  of  the  matching 
process  was  to  develop  a  profile  for  each  of  the  selected  training 
technologies  by  identifying  those  parts  of  the  training  system 
which  they  were  intended  to  enhance.  In  other  words,  each 
technology  is  intended  to  add  to  the  knowledge  or  technology  base 
of  one  or  more  of  the  ISD  processes. 

The  term  "training  technology,"  as  it  is  used  here,  requires 
some  explanation.  Rousseau  defined  technology  as  "a  process  for 
transforming  physical  and  information  inputs  into  outputs  (1983, 
p.  225).”  As  such,  a  training  technology  can  refer  to  a  wide 
range  of  items,  including  hardware,  software,  research 
procedures,  data  collection  instruments ,  and  analysis  tools.  In 
this  study,  researchers  were  primarily  concerned  with  research 
projects  and  not  necessarily  individual  training  technologies, 
per  se.  These  projects  may,  in  fact,  develop  and  integrate 
several  types  of  training  technologies  into  one  system  in  order 
to  achieve  a  specific  goal  or  meet  a  specific  need.  For 
simplicity  sake,  however,  a  project  will  be  referred  to  as  a 
training  technology,  whether  it  encompasses  a  single  technology 
or  a  system  of  integrated  technologies. 

The  most  reliable  sources  of  information  about  the  training 
technologies  were  project  managers  and  laboratory  technical 
directors.  These  people  possessed  the  most  accurate  and  up-to- 
date  knowledge  about  the  specific  projects  and  could  identify  the 
specific  training  functions  that  the  technologies  were  designed 
to  enhance.  Researchers  provided  the  laboratory  technical 
directors  who  were  responsible  for  technical  training-related 
research  with  the  ISD-technology  matrix  seen  in  Figure  4  and  asked 
them  to  indicate  below  each  of  their  technologies  the  specific 
training  processes  being  enhanced  (i.e.,  the  knowledge  or  technical 
base  of  that  ISD  process  is  increased).  They  also  received  a  list 
of  definitions  clarifying  the  function  of  each  ISD  process.  The 
final  set  of  profiles  (Figure  4)  represents  the  collaborative 
efforts  of  the  technical  directors  and  the  project  managers  who 
oversaw  the  research  and  development  of  the  individual 
technologies . 

The  technology  profiles  presented  here  portray  the  intended 
final  form  of  the  technologies.  Since  these  technologies  are 
currently  under  development,  their  profiles  may  or  may  not  change 
with  time.  For  example,  the  Advanced  On-The-Job  Training  System 
( AOTS )  evolved  into  a  production  ready  technology,  the  Base 
Training  System,  which  provides  only  a  portion  of  the  training 
functions  of  the  original  technology  prototype.  It  went  from 


Figure  4.  Profiles  Cataloguing  the  ISD 
Processes  Enhanced  By  Each 
Training  Technology 
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DENOTE8  THAT  A  PROJECT  A0D8  TO  THE  TECHNOtOOY  OR  KNOWLEDGE  BASE  OF  A  PR0CE88 
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DENOTES  THAT  A  PROJECT  A0D8  TO  THE  TECHNOLOGY  OR  KNOWLEDGE  BASE  OF  PROCESS 


(Figure  4  continued) 


DENOTES  THAT  A  PROJECT  ADOS  TO  THE  TECHNOLOOY  OR  KNOWLEDGE  SASE  OF  A  PROCESS 
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being  an  integrated  management ,  evaluation,  development  and 
delivery  system  for  OJT  to  a  management-oriented  system. 
Nevertheless,  the  profiles,  as  presented  here,  provide  a  snap-shot 
of  the  current  direction  of  AF  training  technologies. 

Training  System  Problem  Profiles 

Technology  profiles  represented  one  side  of  the  equation 
in  the  matching  process.  On  the  other  side  were  the  profiles  for 
the  training  system  problems.  Their  purpose  was  to  systematically 
document  those  functions  of  the  training  system  which  researchers 
suspected  to  be  deficient  based  on  their  analysis  of  unit-level 
training  requirements  and  capabilities  (see  Table  6  a  summary  of 
the  training  problems).  The  same  ISO  analytic  framework  used  by 
technical  directors  and  project  managers  to  describe  their 
technologies  were  also  used  by  researchers  to  translate 
descriptions  of  training  system  problems  into  sets  of  specific 
training  system  needs.  The  results  of  the  analyses  are  shown  in 
Figure  5.  Solid  circles  represent  processes  where  evidence 
strongly  suggests  an  ISD  process  is  being  inadequately  performed  at 
the  unit  level  and  thus  needs  improvement.  Empty  circles  represent 
ISD  processes  where  evidence  inconclusively  suggests  improvements 
are  needed. 

Following  are  the  rationale  behind  construction  of  each  of 
the  problem  profiles: 

1.  Inappropriate  Training  Requirements.  This 
situation  strongly  indicated  a  deficiency  in  the  ISD  process 
entitled  "select  tasks  requiring  training."  Naintainers 
felt  that  some  of  the  globally  imposed  training 
requirements,  such  as  block  training,  were  entirely 
inappropriate  or  overtrained.  They  also  felt  that  there  was 
an  absence  or  lack  of  training  in  others  areas,  such  as 
CAMS,  troubleshooting,  basic  system  knowledge,  and  Chemical 
Warfare.  Moreover,  individuals  at  the  unit- level  suffered 
because  managers  and  supervisors  were  uncomfortable  or 
unable  to  make  decisions  on  training  content  for  any 
individual.  This,  in  turn,  negatively  impacted  the 
"management  of  training  requirements"  at  the  units.  The 
situation  described  here  strongly  suggests  that  the 
mechanisms  for  selecting  tasks  requiring  training,  at  the 
unit-level  and  higher,  were  either  flawed  or  unavailable. 

This  situation  may  also  reflect  deficiencies  in  the 
processes  for  "developing  task  listings”  and  "analyzing  job 
tasks."  For  example,  an  inaccurate  job  task  listing  can 
lead  to  inappropriate  training  if  it  includes  irrelevant 
tasks  or  to  a  lack  of  training  if  it  excludes  relevant 
tasks.  Moreover,  if  tasks  were  not  properly  analyzed,  they 
could  lead  to  over-  or  under -training.  Given  the  level  of 
data  and  primitive  state  of  the  framework,  however,  these 
conclusions  are  speculative  at  best. 
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EVIDENCE  PROVIDES  SOME  SUPPORT  FOR  A  DEFICIENCY  IN  PROCESS  (INCONCLUSIVE) 


2.  Lack  of  Qualified  Instructors  and  OJT  Trainers.  This 
situation  was  largely  a  resource  problem ,  in  terms  of  both 
quantity  and  quality.  The  lack  of  available  instructors 
(i.e.,  the  quantity  problem)  indicated  deficiencies  in  the 
ability  to  "select  appropriate  training  settings"  and 
"forecast  resource/logistic  requirements.”  On  the  one  hand, 
there  was  a  failure  to  adequately  forecast  and  assign  the 
necessary  number  of  instructors  to  FTDs,  MATS,  and  units 
(although  this  may  have  been  a  function  of  external 
constraints  such  as  manpower  shortages).  On  the  other  hand, 
this  situation  indicates  that  training  was  inappropriately 
allocated  to  training  settings  with  insufficient  resources. 
This  situation  may  also  indicate  an  inability  at  the  unit 
level  to  properly  "schedule  resources"  for  training. 

When  instructors  were  available,  many  of  them  appeared 
to  lack  technical  competency  and/or  teaching  abilities.  The 
latter  situation  strongly  indicates  that  a  problem  existed 
in  the  "selection  training  requirements"  for  instructors. 
This  conclusion  is  further  supported  by  the  apparent  lack  of 
available  courses  on  the  subject  of  teaching.  However,  the 
evidence  surrounding  the  former  problem,  i.e.,  lack  of 
technical  competency,  is  for  the  most  part  unclear  as  to  its 
implications.  It  may  indicate  potential  deficiencies  in  a 
number  of  training  processes,  but  further  investigation 
would  be  required  to  determine  which  ones  resulted  in  the 
lack  of  technical  competence  amongst  some  instructors. 

3.  Manpower  and  Personnel  Policies.  This  situation  does  not 
implicate  any  specific  training  process.  It  is,  in  fact,  an 
external  factor  (situational  constraint)  which  adversely 
affects  the  training  system  by  increasing  training 
requirements . 

4.  Enhanced  Hands-On  Training/OJT .  This  situation  is  quite 
broad  in  its  implications,  suggesting  improvement  to  all 
aspects  of  OJT  including  planning,  management,  evaluation, 
development  and  delivery.  The  emphasis,  however,  is  on 
"delivery"  and  the  need  for  "instructional  methods"  and 
"instructional  media"  which  accentuate  hands-on  skill 
training.  It  is  unclear  from  the  data  which,  if  not  all,  of 
the  nubprocesses  in  the  selection  of  instructional  methods 
and  media  are  implicated  by  this  problem. 

5.  Lack  of  Training  Opportunities  Due  To  Improved  Weapon 
System  Technology.  This  situation  clearly  indicates  the 
need  for  instructional  developers  to  "select  instructional 
media"  which  will  enable  maintainers  to  train  on  tasks  which 
infrequently  occur  because  of  extremely  high  weapon  system 
reliability  rates.  This  also  indicates  a  need  for  an 
improved  "training  requirements  selection"  process  so  that 
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infrequently  performed  task  are  identified  for  recurring 
training. 

6.  Limited  Training  Time  Due  To  Operational  Commitments. 
This  situation  is  quite  complex  and  indicates  areas  needing 
improvement  within  training  and  external  to  it.  External 
factors  which  limit  time  include  mission  requirements  and 
limited  manpower.  The  implication  for  training  is  to 
increase  the  efficiency  and  availability  of  training  through 
enhanced  "selection  of  instructional  methods”  and  "media" 
and  to  make  "delivery"  of  training  more 
accessible/convenient . 

7.  Lack  of  Enroute  Training.  The  circumstances  surrounding 
this  situation  point  to  problems  in  the  assignment  process 
and  deficiencies  in  the  "scheduling  of  students."  According 
to  interviewees,  the  procedures  for  scheduling  students  for 
enroute  training  were  slow  and  unreliable. 

8.  Inadequate  Feedback  Mechanisms  For  Evaluating  Training 
Courses.  This  situation  indicates  a  need  for  enhanced 
feedback  mechanisms  to  support  the  "maintenance  and  update 
of  the  instructional  system"  and  the  "evaluation  of  the 
external  validity  of  the  instructional  product . " 

Maintainers  had  mixed  impressions  about  whether  or  not  their 
inputs  regarding  the  content  of  training  courses  were  being 
received  and,  when  received,  used  by  course  developers  to 
evaluate  and  update  their  courses. 

9.  Inadequate  Job  Performance  Measurement  Procedures.  This 
situation  points  to  deficiencies  in  the  "development  of  job 
performance  testing  procedures".  Maintainers  claimed  that  QA 
evaluations  were  unstandardized  and  inconsistent  across 
evaluators.  Moreover,  many  of  the  maintainers  and  some  QA 
personnel  felt  that  the  evaluations  were  not  a  good  measure  of 
a  unit '8  capabilities  since  the  same  set  of  personnel  seemed 
to  be  evaluated  every  time  QA  visited  a  unit.  For  example, 
personnel  on  swing  shifts  rarely  were  evaluated.  QA 
evaluators  were  also  criticized  for  not  being  proficient 

in  the  tasks  they  evaluated.  This  problem,  however,  seemed 
to  be  a  temporary  one  brought  about  by  Rivet  Workforce 
training  requirements  and  should  dissipate  once  QA  personnel 
become  more  familiar  with  the  rivetized  tasks.  (QA  personnel 
also  performed  evaluations  of  student  performance  in 
conjunction  with  AMQP.  However,  it  is  unclear  from  the  data 
whether  or  not  these  training  evaluations  suffered  from  the 
same  lack  of  standardization  as  did  the  job  performance 
evaluations . ) 

10.  Lack  of  TDY  Funding.  This  situation  is  predominantly  a 
reflection  of  recent  budget  cutbacks.  It  is  likely, 
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however,  that  training  managers  were  unable  to  forecast  the 
resource,  cost,  and  performance  impacts  of  their  decisions 
and,  therefore,  provided  relied  on  OJT  rather  than  sending 
someone  TDY  to  school.  This,  in  turn,  indicates  training 
settings  were  inappropriately  selected  (i.e.,  selection  by 
default).  For  example,  training  someone  on  a  certain  task 
or  set  of  tasks  in  OJT  may  be  more  costly  in  terms  of  labor 
hours  and  resources  than  sending  that  person  TDY. 

11.  Implementation  Problems  with  Mew  Training  Media.  This 
situation  pertains  to  IVD  technology,  in  particular,  and 
indicates  that  resources  were  not  properly  identified  and 
provided  to  support  IVD  use  and  implementation  at  the 
unit-level.  Moreover,  training  personnel  did  not  know  when, 
where,  or  how  to  use  that  particular  media - 

The  interview  data  and  findings  from  the  content  analysis 
strongly  support  the  existence  of  deficiencies  in  a  large  number  of 
training  processes.  These  deficiencies  represent  deficiencies 
with  the  unit-level  training  system  as  a  whole  and  are  not 
necessarily  confined  to  the  units  themselves.  For  example,  many  of 
the  processes  at  the  front-end  of  the  framework  (i.e.,  "Analyze 
System  Requirements";  "Define  Education  and  Training 
Requirements";  "Develop  Tests  and  Objectives";  "Plan,  Develop  and 
Validate  Instruction" )  are  often  accomplished  by  organizations 
outside  the  unit-level.  The  results  of  these  activities, 
however,  are  nonetheless  intimately  linked  to  the  unit-level 
training  system.  It  is  important  to  understand  unit-level 
training  as  a  system  and  not  as  an  isolated  activity. 

Table  7  contains  a  summary  of  the  specific  ISD  processes 
implicated  by  the  problems  described  in  previous  chapters.  Only 
those  processes  which  are  strongly  supported  by  the  data  are 
listed  here.  The  degree  to  which  each  of  the  processes  is 
deficient  is  unclear  since  no  metric  existed  for  determining  the 
severity  of  each  problem.  Based  on  the  number  of  problem 
descriptions  (Table  6)  which  implicated  a  particular  process,  it 
would  appear  that  "Selection  of  Tasks  Requiring  Training", 
"Selection  of  Instructional  Media",  "Selection  of  Instructional 
Methods",  and  "Delivery  of  Training"  are  the  processes  with  more 
pronounced  deficiencies.  However,  such  a  finding  would  be 
extremely  tentative. 

The  results  of  this  analysis  should  be  interpreted  cautiously 
for  several  reasons.  First,  it  is  likely  that  other  researchers 
would  have  differing  interpretations  of  the  problems  described  in 
this  paper,  and  therefore,  differing  profiles  and  conclusions. 
Second,  since  instructional  development  is  an  integrated  process, 
it  becomes  difficult  to  implicate  any  single  process  as  being 
deficient.  Problems  may  result  from  weaknesses  in  several 
processes,  or  the  deficiencies  in  one  process  may  overshadow  or 
mask  deficiencies  in  others.  Finally,  the  results  are  limited  by 
the  specificity  and  scope  of  the  Phase  I  SLT  data.  Researchers 
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developed  the  analytic  framework  used  in  this  analysis  after  data 
collection  had  already  been  completed.  Consequently,  this 
framework  did  not  guide  the  data  collection  effort.  Data 
collection  was  instead  guided  by  a  similar,  yet  much  broader. 

Table  7 

Summary  of  Training  Processes  Implicated  bv  SLT  Phase  I  Findings 


1)  Selection  of  Tasks  Requiring  Training; 

2)  Selection  of  Appropriate  Training  Settings; 

3)  Forecast  of  Resource/Logistics  Requirements; 

4)  Development  of  Job  Performance  Testing  Standards; 

5)  Selection  of  Instructional  Methods; 

6)  Selection  of  Instructional  Media; 

7)  Delivery  of  Instruction; 

8)  Scheduling  of  Students; 

9)  Scheduling  of  Resources; 

10)  Management  of  Individual  Training  Requirements; 

11)  Maintenance  and  Update  of  Instructional  Systems;  and, 

12)  Evaluation  of  External  Validity  of  Instructional  Product. 


framework  than  the  ISD  model  provided.  This  limited  the  ability  of 
researchers  to  identify  accurately  the  specific  training  system 
needs  implied  by  the  data.  They  were,  for  example,  unable  to 
identify  the  specific  ISD  processes  within  media  and  method 
selection  implicated  by  the  problem  descriptions.  This  inability 
was  largely  due  to  the  lack  of  specificity  in  the  data  collected. 
Such  a  situation  could  have  been  avoided  if  data  collection  had 
been  focused  along  the  lines  of  the  ISD-based  analytic  framework. 

Training  Needs-Technoloov  Match 

The  final  step  in  the  matching  process  involved  a  comparison 
of  the  technology  and  training  system  deficit  profiles.  The 
purpose  of  such  a  comparison  is  to  reveal  where  training  system 
needs  are  being  addressed  by  AF  training  research  and  where  they 
are  not.  The  results  of  the  matching  process  can  be  seen  in 
Figure  6.  The  shaded  areas  represent  processes  which  were 
implicated  as  being  deficient  from  earlier  analyses.  An  overlap 
(or  "hit")  exists  when  solid  circle  appears  in  the  shaded  area. 
This  indicates  that  the  deficient  ISD  process  is  addressed  by  a 
training  technology.  The  mere  fact  that  an  overlap  exists, 
however,  is  not  confirmation  in  and  of  itself  that  a  system  need 
is  being  addressed.  Since  the  analytic  framework  used  in  this 
matching  process  is  very  basic  and  only  considers  function, 
researchers  have  to  consider  other  issues  such  as  the  context 
surrounding  a  training  need  (domain)  and  the  degree  to  which  a 
technology  addressed  this  context  in  addition  to  the  basic  overlap. 
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Figure  6.  Training  Technology  and 
Training  System  Needs  Match 
Indicating  Areas  For  Potential 
Application 
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OENOTE8  THAT  A  PROJECT  AODS  TO  THE  TECHNOLOGY  OR  KNOWLEDGE  BASE  OF  PROCESS 


(Figure  6  continued) 
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DEMOTES  THAT  A  PROJECT  ADDS  TO  THE  TECHNOLOGY  OR  KNOWLEDGE  BASE  OF  A  PROCESS 


The  results  of  this  matching  process  are  discussed  in 
the  remainder  of  this  section,  focusing  on  each  of  the  processes 
listed  as  deficient  in  Table  7  and  their  relationship  to  AF 
training  technology  research. 

The  ISD  process  entitled  "select  tasks  requiring 
training"  was  implicated  as  being  deficient  several  times  in  the 
analysis  of  training  system  problems.  Figure  6  indicates  that 
four  technologies  (i.e.,  projects)  enhance  the  knowledge  or 
technology  base  for  the  selection  of  training  requirements, 
including  the  Advanced  OJT  System  (AOTS)/BTS,  the  Comprehensive 
Occupational  Data  Analysis  Package  (CODAP),  Job-Aiding/Training 
Allocation  Technologies  (JATAT),  and  Training  Evaluation.  Each 
of  these  technologies  is  intended  to  enhance  the  training 
requirements  selection  process  in  a  slightly  different  manner. 
Unfortunately,  the  analytic  framework,  as  it  is  currently 
defined,  does  not  distinguish  between  these  subtle  differences. 
Even  if  it  could,  the  level  of  specificity  of  the  data  would 
preclude  a  more  in  depth  comparison  of  needs  and  technologies. 
Suffice  it  to  say  that,  based  on  the  interview  data,  resources 
for  accurately  selecting  training  tasks  need  to  exist  at  all 
levels  of  the  training  system.  This  will  enable  training 
managers  to  tailor  training  requirements  to  their  particular 
situations . 

Deficiencies  in  the  processes  for  "selecting  appropriate 
training  settings"  and  "forecasting  resource  logistic 
requirements"  appear  to  be  closely  related.  The  reason  being 
that  forecasting  resource  and  logistic  requirements  was  an 
important  step  in  determining  appropriate  training  settings. 

In  other  words,  the  appropriateness  of  a  training  setting  was 
largely  dependent  upon  the  adequacy  of  its  resources.  Three  of 
the  training  technologies,  the  AOTS,  CBT  Selection  Advisor,  and  the 
Training  Decisions  Modeling  Technologies  ( TMDT ) ,  provide  a 
capability  for  forecasting  resource  and  logistic  requirements. 
However,  only  one  of  these  technologies,  the  TDMT,  goes  the 
additional  step  and  enhances  the  process  for  selecting 
appropriate  training  settings  based  on  the  forecasted  resource 
and  logistic  requirements. 

At  a  very  broad  level,  the  process  for  "selecting 
instructional  methods"  was  implicated  by  two  of  the  apparent 
training  system  problems.  Several  technologies,  including  the 
Advanced  Instructional  Design  Advisor  (AIDA),  Basic  Job  Skills 
(BJS ) ,  CBT  Selection  Advisor,  and  Training  Evaluation,  address 
some,  if  not  all,  aspects  of  this  deficiency.  Unfortunately,  as 
previously  stated,  researchers  could  not  discern  from  the  data 
which  of  the  subprocesses  needed  improvement,  and 
therefore,  it  is  unclear  which  technologies  truly  address  the 
deficiency.  The  only  exception  to  this  would  be  AIDA  which 
addresses  each  aspect  of  method  selection. 

The  CBT  Selection  Advisor  is  the  only  technology  which 
addresses  the  deficiencies  in  "selecting  instructional  media". 

It  would  appear  that  this  technology  fully  addresses  the 
deficiencies  in  media  selection,  but  appearances  can  be 
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deceiving.  The  overlaps  shown  in  Figure  6  simply  indicate  that  a 
technology  enhances  the  same  type  of  functions  which  have  been 
identified  as  deficient.  A  closer  examination  of  the  situation 
reveals  that  the  technology  in  question  is  intended  to  enhance 
media  selection  and  use  within  one  specific  context  —  computer- 
based  training.  The  deficiencies  indicated  by  the  Phase  I 
research  appear  to  encompass  media  selection  at  a  more  general 
level;  that  is,  it  is  not  content  specific.  Training  developers 
and  manager  need  help  identifying  training  media,  CBT  or  other, 
suitable  to  the  unit-level  environment.  Whether  or  not  the  CBT 
Selection  Advisor  can  be  adapted  for  media  selection  in  general 
is  unknown. 

Deficiencies  in  the  "determination  of  resource  and  funding 
requirements”  for  IVD  instruction  contributed  to  the  poor 
implementation  of  this  category  of  instruction  at  units.  The 
AOTS,  CBT  Selection  Advisor,  and  TDMT  are  possible  solutions  to 
this  deficiency,  however.  These  technologies  would  enable 
training  developers  to  determine  the  exact  resource  and  funding 
requirements  necessary  to  support  their  instruction  prior  to 
actually  implementing  it. 

Training  delivery  is  the  most  salient  process  in  the 
instructional  system  and  is  the  culmination  of  all  the  other 
processes.  It  is,  if  you  will,  "where  the  rubber  meets  the 
road."  Of  the  fifteen  training  technologies  of  concern  in  this 
study,  almost  half  (i.e.,  seven)  of  them  are  intended  to  enhance 
the  delivery  of  instruction.  They  do  so  through  a  variety  of 
computer-based  and  intelligent  tutoring  systems.  However, 
deficiencies  which  appear  in  the  delivery  process  are  often  the 
result  of  deficiencies  in  other  processes.  While  the  interview 
data  suggests  that  the  delivery  of  instruction  needs  improvement 
to  overcome  problems  in  the  unit-level  environment  such  as 
limited  training  time,  it  is  likely  that  the  delivery  problem 
would  be  solved  if  other  processes  (e.g.,  selection  of 
instructional  media)  were  improved. 

All  but  one  of  the  processes  within  the  instructional  support 
area  were  identified  as  deficient.  These  processes,  along  with 
instructional  delivery  and  evaluation,  are  perhaps  the  most 
germane  to  operational  units  (as  opposed  to  the  training  planning, 
programming  and  development  functions  which  typically  occur 
outside  the  units  themselves ) .  According  to  the  data  collected 
at  PACAF  F-16  maintenance  units,  the  scheduling  of  both  resources 
(i.e.,  instructors)  and  students  (i.e.,  for  enroute 
training)  was  deficient.  The  AOTS/BTS  and  Instructional  Support 
System  (ISS)  are  intended  to  address  this  type  of  deficiency.  In 
fact,  the  AOTS/BTS  incorporates  ISS  technology  into  itself. 

Deficiencies  in  the  "management  of  individual  training 
requirements"  also  degraded  the  instructional  support  capabilities 
of  unit-level  training.  This  situation  is  very  much  related  to  the 
problem  surrounding  "selection  of  tasks  requiring  training"  and 
may,  in  fact,  only  be  a  consequence  of  this  deficiency.  In  this 
case,  however,  training  managers  were  unable  or  unwilling  to 
tailor  training  requirements  to  the  needs  of  individuals  at  a 
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job-site  as  opposed  to  the  needs  of  an  AF  specialty  or  one  of  its 
shreds.  The  AOTS,  once  again  purports  to  enhance  the  process  for 
managing  individual  training  requirements.  It  is  also  likely  that 
the  capabilities  found  in  CODAP,  JATAT,  and  Training  Evaluation 
are  applicable  to  this  process  (just  as  they  are  to  "selection  of 
tasks  requiring  training"). 

Instructional  support  also  suffered  from  deficiencies  in  the 
"maintenance  and  update  of  the  instructional  system. "  More 
specifically,  interviewees  felt  that  training  courses  were  not 
being  updated  or  modified  based  on  feedback  from  the  field  (via 
follow-up  surveys,  course  critiques,  and  informal 
communications).  Several  technologies  offer  potential  solutions 
to  this  problem  by  strengthening  the  feedback  loop  between  the 
field  and  the  course  developers,  instructors  and  administrators. 
These  technologies  include  AOTS,  CODAP,  and  the  Integrated 
Maintenance  Information  System  ( IMIS ) .  Whether  or  not  the 
information  provided  by  these  technologies  is  actually  used  to 
update  courses  is  a  separate  question  beyond  the  scope  of  this 
research. 

Finally,  the  deficiencies  in  the  feedback  mechanisms  just 
described  also  indicate  deficiencies  in  the  process  for 
"evaluating  the  external  validity  of  instruction."  The  fact  that 
some  interviewees  felt  courses  were  not  being  updated  to 
reflect  the  realities  of  the  operational  environment  also  indicates 
that  the  external  validity  of  some  courses  is  questionable  (at 
least  in  terms  of  their  face  validity).  The  AOTS  and  JPM 
technologies  are  intended  to  improve  this  process  in  the  training 
system.  The  information  generated  by  these  technologies  should 
allow  course  developers  to  evaluate  the  degree  to  which  their 
courses  teach  the  skills  and  knowledges  relevant  to  the  work 
environment  including  appropriateness  of  course  content  and 
transfer  of  training. 

D.  Discussion  of  Matching  Process 

At  first  glance,  there  would  appear  to  be  no  apparent  gaps 
in  the  AF  training  technology  research.  Each  of  the  ISD  processes 
that  researchers  found  to  be  deficient  in  the  unit-level  training 
system  for  PACAF  F-16  maintenance  units  are  in  some  way  addressed 
by  a  training  technology.  The  only  area  which  may  be  overlooked 
is  that  of  training  media  selection.  Currently,  only  one 
technology,  the  CBT  Selection  Advisor,  addresses  this  problem, 
but  it  does  so  from  a  CBT  perspective  only. 

The  training  system  needs  and  technology  match  also  reveals 
areas  where  technologies  are  focused  but  which  were  not  implicated 
by  the  data  as  being  deficient.  Bowever,  this  does  not  mean  that 
the  technologies  are  "misguided."  First  of  all,  this  study  was 
limited  to  PACAF  F-16  maintenance  units.  The  training 
technologies,  on  the  other  hand,  are  being  developed  for  a  broader 
AF  application.  Second,  while  only  certain  processes  were 
implicated  as  deficient  at  the  unit  level,  improvements  made  to  any 
of  the  processes  are  potentially  beneficial  to  unit-level  training. 
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Finally/  the  data  collection  instruments  used  in  this  study  were 
not  based  on  the  analytic  framework  as  seen  in  figures  4,  5,  and  6. 
Consequently/  there  is  some  degree  of  error  in  the  results 
presented  here. 

Moreover,  the  framework  presented  in  this  report  has  a 
number  of  limitations  which  should  also  be  acknowledged.  It  was 
developed  specifically  for  describing  and  relating  training 
technologies  to  training  system  needs  at  a  very  basic  level;  that 
is,  it  merely  shows  where  the  functions  of  training  technologies 
match  deficiencies  in  the  training  system.  The  framework  does 
not,  for  instance,  specify  the  theoretical  principles, 
procedures,  and  rules  for  developing  and  operationalizing 
constructs,  and  for  analyzing  and  interpreting  data  (see  Ostroff 
&  Ford,  1989).  Nor  does  the  framework  explicitly  address  the 
issue  of  training  domains;  i.e.,  the  specific  contextual  areas 
with  which  training  technologies  or  deficits  are  associated,  such 
as  aircrew,  maintenance,  and  the  like. 

Nevertheless,  the  analytic  framework  represents  an 
important,  although  preliminary  step  toward  development  of  a 
systematic  process  for  identifying  and  describing  training  system 
needs,  cataloguing  training  technology  functions,  and  relating  the 
two.  Future  work  in  this  area  would  greatly  benefit  from  the 
theoretical  developments  found  in  training  needs  assessment 
research.  Ostroff  and  Ford  (1989),  for  example,  presented  a 
model  of  needs  assessment  based  on  a  levels  perspective. 

According  to  this  model,  training  needs  occur  at  three  levels: 
organizational,  subunit,  and  individual.  Each  of  these  levels  of 
analysis  has  implications  for  construct  development, 
operationalization,  and  interpretation.  It  is  likely  that  these 
concepts  would  also  be  applicable  to  training  systems  needs 
assessment  should  research  continue  in  this  area. 

In  summary,  the  data  indicates  that  training  system  needs 
are  being  addressed  by  training  technologies.  These 
relationships,  however,  deserve  a  more  in  depth  examination. 
Researchers  must  go  beyond  the  question  of  whether  or  not 
technologies  are  functionally  matched  to  deficiencies.  They  must 
examine  issues  such  as  the  degree  to  which  a  technology  meets  a 
need,  the  seriousness  of  a  training  system  need,  and  the  context 
surrounding  the  deficiencies.  Such  an  examination  would  reveal 
specific  areas  where  training  technologies  could  be  focused  in 
order  to  increase  their  applicability  to  unit-level  training. 
Before  this  can  take  place,  researchers  must  first  advance  the 
framework  which  has  guided  the  analysis  described  in  this 
chapter. 
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VI.  CONCLUSIONS 


Phase  I  of  the  Squadron  Level  Training  research  effort 
provided  researchers  with  a  greater  understanding  of  the  unit- 
level  training  environment  and  the  potential  for  training 
technology  applications  within  PACAF  F-16  maintenance  units. 

It  identified  trends  in  current  training  requirements,  evaluated 
the  training  capabilities  at  the  unit-level,  identified  factors 
facilitating  or  inhibiting  training,  and  provided  a  preliminary 
match  between  technologies  and  problem  areas  within  the  overall 
unit-level  training  system.  However,  the  primary  benefits  derived 
from  this  initial  phase  of  research  were  the  methodologies  and 
future  directions  suggested  by  it.  This  information  should  form 
the  basis  for  a  more  comprehensive  follow-on  study. 

Findings  from  Phase  I  suggest  that  units,  at  least  PACAF  F- 
16  maintenance  units,  are  facing  a  complex  training  environment. 
Fluctuations  in  experience  levels  amongst  maintenance  personnel, 
advances  in  weapon  system  technology,  and  policies  such  as  Rivet 
Workforce  have  increased  training  requirements  for  maintenance 
units.  At  the  same  time,  the  data  suggests  that  the  unit-level 
training  system  has  numerous  deficiencies.  These  deficiencies 
exist  at  all  levels  within  the  training  system,  from  the 
selection  of  tasks  requiring  training  to  the  selection  of 
instructional  media  to  the  evaluation  of  instructional  products. 
Fortunately,  it  appears  as  though  AF  training  technologies 
address,  to  a  greater  or  lesser  extent,  each  the  deficiencies 
identified  in  this  research. 

While  this  research  provides  some  insight  into  unit-level 
training,  findings  from  this  research  are  restricted  to  a  small 
segment  of  the  AF;  namely,  PACAF  F-16  maintenance  units.  As  such, 
these  findings  have  limited  generalizability .  The  strength  of 
the  Phase  I  research,  however,  is  not  limited  to  its  analysis  of 
F-16  maintenance  units,  per  se.  Instead,  this  research  provides 
the  AF  with  suggestions  for  the  continued  study  of  unit-level 
training. 

Interviews  with  trainees,  trainers,  and  training  managers 
and  the  associated  content  analyses  resulted  in  a  number  of 
valuable  lessons  for  researchers.  First,  it  reinforced  the  fact 
that  training  exists  as  a  system.  At  the  unit  level,  personnel 
engage  in  a  variety  of  training  activities  including  the  day-to- 
day  management  of  training,  training  delivery,  and  training 
evaluation.  Other  activities  such  as  training  planning  and 
development,  while  typically  occurring  outside  the  unit, 
significantly  impact  unit-level  training  nonetheless. 
Consequently,  future  research  should  continue  to  examine  each 
aspect  of  the  training  system  as  it  relates  to  unit-level 
training. 

Second,  the  information  gathered  during  data  collection  in 
this  phase  will  help  guide  the  development  of  surveys  in  the 
next  effort.  This  information  has  provided  researchers  with 
insight  into  the  types  of  data  available  at  the  unit  level  and  the 


type 8  of  information  unit  personnel  are  capable  of  providing  or 
understanding.  For  example ,  researchers  found  that  very  few 
maintenance  personnel  knew  what  an  IVD  system  was. 

Finally,  Phase  I  resulted  in  the  development  of  an  ISD-based 
framework  for  analyzing  training  technologies  and  training  system 
needs,  and  then  relating  the  two.  This  framework,  although  only 
preliminary,  provides  an  excellent  structure  for  guiding  and 
focusing  future  data  collection  and  analysis.  Moreover,  it  lays 
the  foundation  for  development  of  a  more  versatile  tool  which  can 
be  used  to  evaluate  the  application  of  training  technologies  to 
the  unit-level  training  system  in  terms  of  its  appropriateness, 
cost /benefits,  feasibility,  and  other  criteria  (see  Appendix  G  for 
a  list  of  potential  evaluative  criteria ) .  The  outcome  from 
such  an  analysis  will  be  a  Science  &  Technology  Plan  for  guiding 
the  development  and  implementation  of  HSD  training  technologies. 

In  an  environment  of  increasing  training  requirements  and 
decreasing  dollars,  training  research  must  take  the  most  prudent 
and  expedient  route.  The  research  described  in  this  paper 
represents  an  initial  step  toward  understanding  and  evaluating 
the  application  of  training  technologies.  However,  this  research 
must  be  refined  and  extended  to  enlisted  personnel  in  all  AF 
support  units  in  order  to  provide  AF  training  technology 
researchers  with  information  about  the  most  promising 
opportunities  for  the  application  of  training  technologies. 
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APPENDIX  A:  GLOSSARY 


LIST  OF  ACRONYMS 

ABDR  Aircraft  Battle  Damage  Repair 

AFETS  Air  Force  Engineering  and  Technical  Services 

AFHRL  Air  Force  Human  Resources  Laboratory 

AFJQS  Air  Force  Job  Qualification  Standard 

AFMPC  Air  Force  Military  Personnel  Center 

AFS  Air  Force  Specialty 

AFSC  Air  Force  Specialty  Code 

AGS  Aircraft  Generation  Squadron 

AIDA  Advanced  Instruction  Design  Advisor 

AL/HR  Human  Resources  Directorate  of  the  Armstrong  Laboratory 

(formerly  the  Air  Force  Human  Resources  Laboratory) 

AMQP  Aircraft  Maintenance  Qualification  Program 

AMU  Aircraft  Maintenance  Unit 

AOTS  Advanced  On-the-Job  Training  System 

(now  called  Base  Training  System) 

ATC  Air  Training  Command 

ATS  Advanced  Training  System 

BJS  Basic  Job  Skills 

CAMS  Core  Automated  Maintenance  System 

CASB  Combat  Ammunition  System  Base 

CBI  Computer  Based  Instruction 

CBT  Computer  Based  Training 

CDC  Career  Development  Course 

CJQS  Command  Job  Qualification  Standard 
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COMO 


CRS 

CUT 

DCM 

EMS 

EST 

FTD 

HQUSAF 

HSD 

ICATT 

IMIS 

ISD 

ISS 

ITT 

IVD 

JATAT 

JQS 

MAJCOM 

MAT 

MI 

MTBF 

MTL 

MTT 

MTTR 

NAF 

NCO 


Combat  Oriented  Maintenance  Organization 
Component  Repair  Squadron 
Cross-Utilization  Training 
Deputy  Commander  for  Maintenance 
Equipment  Maintenance  Squadron 
Enlisted  Specialty  Training 
Field  Training  Detachment 
Headquarters  United  States  Air  Force 
Human  Systems  Division 

Intelligent  Computer-Aided  Training  Testbeds 
Integrated  Maintenance  Information  System 
Instructional  System  Development 
Instruction  Support  System 
Intelligent  Training  Technologies 
Interactive  Videodisc 

Job-Aiding/Training  Allocation  Technologies 
Job  Qualification  Standard 
Major  Command 

Office  symbol  for  branch  of  TMD 
(formerly  Maintenance  Training  [Division]) 

Maintenance  Instructor 

Mean  Time  Between  Failure 

Master  Task  List 

Mobile  Training  Team 

Mean  Time  to  Repair 

Numbered  Air  Force 

Noncommissioned  Officer 
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OJT 

OMC 

OSR 

PACAF 

PCS 

PME 

QA 

RWF 

SEI 

SLT 

SME 

SOW 

STS 

TAC 

TDMT 

TDY 

TMD 

TO 

TOT 

UES 


On-the-Job  Training 
Occupational  Measurement  Center 
Occupational  Survey  Report 
Pacific  Air  Forces 
Permanent  Change  of  Station 
Professional  Military  Education 
Quality  Assurance 
Rivet  Workforce 
Special  Experience  Identifier 
Squadron  Level  Training 
Subject  Matter  Expert 
Statement  of  Work 
Specialty  Training  Standard 
Tactical  Air  Command 

Training  Decisions  Modeling  Technologies 
(TDS  forms  the  baseline  technology) 

Temporary  Duty 

Training  Management  Division 

Technical  Order 

Task  Oriented  Training 

UES,  Inc.  (formerly  Universal  Energy  Systems,  Inc.) 
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LIST  OF  TERMS 


Ancillary  training  Training  for  functional  or  additional  duty 
tasks  done  outside  the  scope  of  the  primary  job  (e.g.,  CPR, 
Code  of  Conduct ) . 

Archival  data  Existing  records  and  information  which  may  be 
accessed  to  provide  details  of  on-going  operations  or 
historical  occurrences . 

Attribute  A  distinguishing  feature  or  component  of  a  training 
technology. 

Block  training  The  consolidation  of  training  requirements  into  a 
block  of  instruction  to  be  presented  in  a  single  training 
session. 

Cross  Utilization  Training  Tasks  outside  the  normal  duty  of  an 
AFSC  that  are  included  in  the  STS  to  provide  flexibility  under 
austere  conditions. 

Development  and  Delivery  All  activities  concerned  with  authoring 
and  delivering  instruction  such  as  test  development, 
curriculum  development,  identification  of  delivery  methods, 
and  delivering  instruction. 

Enroute  training  Training  received  attendant  to  a  permanent  change 
of  station  at  a  location  other  than  the  permanent  duty 
station,  usually  at  an  FTD. 

Homogeneity  analysis  The  degree  to  which  the  units  surveyed  show 
commonality  among  responses  to  the  interview  questions 
regarding  training  needs,  capabilities,  and  so  on. 

Job-related  training  Task-specific  and  technical  training; 
includes  initial /qualification,  continuation,  upgrade, 
requalification,  conversion,  and  other  (e.g.,  FTD) 

Management  and  Evaluation  The  day-to-day  operation  of  training 

including  scheduling  students,  courses,  and  classes;  keeping 
of  individual,  course,  and  class  records;  resource  management; 
and  evaluating  students,  courses,  and  training  systems. 

Operational  level  The  lowest  level  of  the  organization  at  which 
the  actual  day-to-day  production,  maintenance,  and  so  on  gets 
accomplished . 


Planning  and  Programming  All  activities  undertaken  to  plan  or 
program  training  for  Air  Force  personnel.  This  includes  all 
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aspects  of  establishing  instruction  such  as  identifying  the 
who,  what,  when,  and  where  of  training;  budgeting  and  resource 
actions;  and  acquisition  of  training. 


Reliability  The  dependability  of  a  training  device  or  method 
during  use.  This  can  also  be  thought  of  as  "does  the 
equipment  usually  work  or  not?" 

Shredout  An  alphabetical  suffix  attached  to  an  AFSC  to  identify 
specific  equipment,  functions,  or  positions  within  a 
specialty. 

Training  capabilities  The  unit-level  resources  available  for 
training  (e.g.,  time  away  from  primary  duties,  materials, 
equipment,  experienced  personnel  to  serve  as  instructors, 
budget  for  supplies  and  TDY  funding ) . 

Training  environment  Description  of  the  factors  which  influence 
the  provision  of  unit  level  training,  such  as  organizational 
mission,  manning  issues,  training  requirements,  and  training 
capabilities . 

Training  function  Reflects  three  levels  of  involvement  in  the 
training  system:  Planning  and  Programming ,  Management  and 
Evaluation,  and  Development  and  Delivery. 

Training  needs  Specific  processes  within  the  training  system  which 
need  improvement;  these  deficiencies  contribute  to  the 
manifestation  of  observable  training  problems 

Training  problems  General  situations  which  degrade  training;  they 
are  manifestations  of  underlying  deficiencies  in  the  training 
system  and  its  environment 

Training  reguirements  Current  and  future  training  as  specified  by 
AF,  MAJCOM,  and  unit-level  doctrine;  specifications  to  include 
types  of  training,  training  content,  and  amounts  of  training 
to  be  conducted  (e.g.,  number  of  trainees,  length  of  training, 
frequency  of  training). 

Training  system  The  ordered  arrangement  of  processes  for  planning, 
programming,  managing,  evaluating,  developing  and  delivering 
training  within  a  given  environment. 

Training  technologies  Any  process  for  transforming  physical  and 
informational  inputs  into  training  outputs;  may  be  either 
currently  existing  or  in  the  development  process. 

Transfer  opportunities.  The  feasibility  of  relocating  training 
performed  at  central  location  to  the  unit  level,  and  vice 
versa. 
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APPENDIX  B: 


LIST  OF  INFORMATIONAL  INTERVIEWS  BY  OFFICE  SYMBOL 


Headquarters  USAF,  Washington  DC 

LEXY 

LEYM 

LEYW 

LEYX 

DPPT 


Headquarters  Tactical  Air  Command,  Langley  AFB,  VA 
L6M 

LGMF-16 

LGQ 

LGQT 

LGQP 


Headquarters  Pacific  Air  Forces,  Hickam  AFB,  HI 

DPAT 

LGMS 

LGMM 

LGMMR 

LGMFB 

LGWS 

TTA 


Headquarters  Air  Training  Command,  Randolph  AFB,  TX 

TTO 

ACC 


Headquarters  AF  Military  Personnel  Center,  Randolph  AFB,  TX 
DPMRPQ 
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APPENDIX  C:  DATA  COLLECTION  INTERVIEW  PROTOCOL 
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SQUADRON  LEVEL  TRAINING  RESEARCH  INITIATIVE 
STRUCTURED  INTERVIEW 
(MAINTENANCE) 

Good  morning/afternoon/evening.  I'm  .  from  the  Air 

Force  Human  Resources  Laboratory,  Air  Force  Systems  Command.  I'm 
part  of  a  team  here  to  study  research  issues  in  unit  level  training 
specifically  dealing  with  F-16  maintenance  units.  We  would  like  to 
emphasize  that  your  personal  comments  will  remain  strictly 
confidential  and  your  answers  will  be  combined  with  other 
maintenance  responses  in  a  group  summary.  All  of  the  information 
will  be  used  by  AFHRL  and  the  Human  Systems  Division  of  AFSC  to 
help  guide  research  in  unit  level  training  over  the  next  six  to 
twelve  years. 

We  are  going  to  first  ask  you  some  questions  about  yourself  then 
follow  with  som  )  questions  concerning  training  at  the  operational 
unit . 

Do  you  have  any  questions?  ....  Ok  fine,  then  let's  get  started. 


70 


DEMOGRAPHICS 


GRADE  (circle  one): 

2Lt  lLt  Capt  Major  Lt  Col  Col 
Amn  A1C  SRA  Sgt  SSgt  TSgt  MSgt  SMsgt 
MAJCOM  (circle  one): 

PACAF  TAC  HQ  PACAF  HQ  TAC  Air  Staff 

BASE:  _ 

UNIT:  _ 

TIME  IN  UNIT:  _ 

JOB  TITLE:  _ 

DAFSC:  _ 

Previous  Maintenance  AFSC ( s ) :  _ 

TIME  POSITION:  _ 

TOTAL  MAINTENANCE  TIME: 

TOTAL  F-16  MAINTENANCE  TIME:  _ 

UNIT  MISSION: 


CURRENT  ROLE  IN  UNIT  TRAINING  PROCESS: 


CMSgt 
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OPERATIONAL  PERSPECTIVE 


"I  want  to  ask  you  some  questions  about  your  perceptions  and 
experiences  with  training  in  your  unit.  The  first  set  of  questions 
will  f ecus  on  training  requirements . " 

A.  Training 

1.  What  types  of  job  and  ancillary  training  are  currently 
conducted: 

a.  at  the  unit  itself? 

b.  at  FTDs? 

c.  at  MATs? 

d.  at  other  locations? 

2.  Is  there  any  job  or  ancillary  training  currently  being 
conducted  which  is  not  needed? 

3.  Are  there  any  topics  or  tasks  which  are  overtrained? 

4.  Is  there  any  job  or  ancillary  training  currently  not 
being  conducted  which  should  be  trained? 

5.  Are  there  any  topics  or  tasks  which  are  undertrained? 

6.  Is  there  any  training  that  is  currently  conducted  at  the 
unit  level  which  you  feel  should  be  conducted  elsewhere 
( i . e . ,  at  FTDs ,  MATs ,  tech  school ) ? 

Yes: 

a.  Which  training? 

b.  Where  should  it  be  conducted? 

c.  Why? 

7.  Is  there  any  training  that  is  currently  conducted  away 
from  the  squadron  which  you  think  should  be  conducted  at 
the  squadron? 

Yes: 


a.  Which  training? 

b.  Why? 
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8.  Will  training  needs  be  different  in  the  future? 

Yes 

a.  What  will  change? 

b.  What  effect  will  this  have  on  unit  training? 

c.  How  will  this  affect  mission  readiness? 


B.  Planning /Programming 

"This  next  set  of  questions  focuses  on  the  Planning  and  Programming 
of  training.  Planning  and  Programming  includes  such  activities  as 
identifying  the  who,  what,  when,  and  where  of  training;  budgeting 
and  resource  actions;  and  acquisition  of  training.” 

1.  Are  you  involved  in  planning  and  programming  training? 

Yes 

a .  Function? 

_  planning/programming 

_  training  content 

_ training  content  allocation 

_  training  time  allocation 

_  budgeting 

_  resource  planning 

b.  What  specific  products  or  services  do  you 
provide  with  regard  to  your 
planning /programming  function? 


EX:  OPR  for  regulations 
news  letters 
reports 

c.  Do  you  interact  with  any  other  organizations  or 
individuals  when  performing  your  planning/ 
programming  function? 

d.  What  information  do  you  use  in  your  planning/ 
programming  function? 

e.  What  information  would  you  like  to  have  for 
your  planning/programming  function? 
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f .  Do  you  have  any  structured  procedures  for  your 
planning /programming  function? 

EX:  regulations 

locally  prescribed 

procedures 

g.  What  tools  or  methods  do  you  currently  use  to 
accomplish  your  planning /programming  function? 

develop 

evaluate 

update 

h.  Do  you  have  access  to  any  tools  or  methods 
which  are  intended  to  help  you  in  your 
planning /programming  function  but  which  you 
don't  use? 

i.  What  tools  or  methods  would  you  like  to  have  to 
do  your  planning/programming  function? 


develop 

evaluate 

update 

j.  Do  you  foresee  any  future  occurrences  or  events 
which  will  impact  training  planning  and 
programming? 

k.  Are  there  any  other  problems  in  planning  and 
programming  we  have  not  already  discussed? 

l.  Where  do  you  see  the  highest  payoff  with  regard 
to  the  use  of  tools /methods  in  planning  and 
programming? 

m.  Other  persons  who  we  should  talk  to? 

2.  Is  your  unit  given  sufficient  voice  in  tailoring 
training  (the  what,  when,  and  where)  to  unit  specific 
missions? 

3.  Are  there  any  feedback  mechanisms  in  place  which  allow 
you  or  your  unit  to  influence  training? 

Yes: 

a.  What  are  they? 

b.  Are  they  effective? 
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4.  Are  there  any  feedback  mechanisms  you  would  like  to  see 
implemented? 

5.  Are  you  generally  satisfied  with  the  planning  and 
programming  of  training? 

C .  Management 

"These  next  questions  will  focus  on  training  management.  These 
activities  include  the  day  to  day  operations  of  training  such  as 
planning  individual  training;  resource  management;  scheduling 
students,  courses,  and  classes;  and,  keeping  individual,  course, 
and  class  records . " 

1.  Are  you  involved  in  the  day-to-day  management  of 
training? 

Yes 

a.  Function? 

_  individual  training  planning 

_  scheduling  students /courses 

_  resource  management 

_  recordkeeping,  students/ 

courses 


b.  What  specific  products  or  services  do  you 
provide  with  regard  to  your  management 
function? 

EX:  OPR  for  regulations 

news  letters 
reports 

c.  Do  you  interact  with  any  other  organizations  or 
individuals  when  performing  your  management 
function? 

d.  What  information  do  you  use  in  your  management 
function? 

e.  What  information  would  you  like  to  have  for 
your  management  function? 

f.  Do  you  have  any  structured  procedures  for  your 
management  function? 


EX:  regulations 

local  procedures 
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g.  What  tools  or  methods  do  you  currently  use  to 
accomplish  your  management  function? 

develop 

evaluate 

update 

— to  determine  individual  training 
requirements 

— to  track  training 

h.  Do  you  have  access  to  any  tools  or  methods 
which  are  intended  to  help  you  in  your 
management  function  but  which  you  don't  use? 

i.  What  tools  or  methods  would  you  like  to  have  to 
do  your  management  function? 

develcp 

evaluate 

update 

j.  Do  you  foresee  any  future  occurrences  or  events 
which  will  impact  training  management? 

k.  Are  there  any  other  problems  in  management  we 
have  not  already  discussed? 

l.  Where  do  you  see  the  highest  payoff  with  regard 
to  the  use  of  tools/methods  in  training 
management? 

(RESOURCE  m.  What  tools  or  methods  do  you  currently  use  to: 

PLANNING 

ONLY)  —  manage  shared  resources? 

—  identify  resources  requirements? 

—  evaluate  training  capacities? 

—  estimate  TDY  travel  for  students? 

—  prioritize  training  requirements? 

n.  Other  persons  who  we  should  talk  to? 

2.  Is  there  adequate  time  available  for  your  training  at 


—  unit? 
—  FTD? 
—  MAT? 

No: 
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a.  What  are  the  factors  limiting  your  availability 

for  training? 

b.  How  do  these  factors  impact  the  quality  of 
training? 

3.  Does  your  training  load  keep  you  away  from  your  unit's 
mission? 

Yes 


a.  What  types  of  training  are  impeding  performance 

of  your  unit's  mission? 

b.  How  do  they  impede  performance  of  your 
contribution  to  the  unit's  mission? 

4.  Are  trainers  (i.e.,  instructors)  typically  available  to 
conduct  training  when  training  is  needed? 

No: 

a.  Why? 

b.  How  does  this  impact  the  quality  of  training? 

c.  How  would  you  like  training  to  be  delivered? 


5.  Are  the  necessary  resources,  such  as  training  materials, 

equipment,  facilities,  available  for  your  training? 

No 

a.  Why? 

b.  To  what  degree  does  this  affect  training? 

c.  What  types  of  resources  are  typically  not 
available  for  training? 

d.  What  resources  do  you  need  for  training? 

6.  Do  you  know  which  tasks  or  topics  you  require  training 
on? 

—  job  tasks? 

—  ancillary? 

a.  Is  this  information  available  to  you? 

b.  Is  this  information  helpful  to  you? 

c.  In  what  format  do  you  receive  this  information? 

d.  In  what  format  would  you  like  to  receive  this 
inf o. ? 

7.  Are  you  generally  satisfied  with  training  management? 
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D.  Evaluations 


"These  next  questions  focus  on  evaluations .  This  includes 
evaluation  of  students,  training  courses,  and  the  training  system 
itself . " 

1.  Are  you  involved  in  evaluations? 

Yes 

a.  Function? 

_  student  evaluations 

_  course  evaluations 

_  system  evaluations 

b.  What  specific  products  or  services  do  you 
provide  with  regard  to  your  evaluation 
function? 

EX:  OPR  for  regulations 

news  letters 
reports 

c.  Do  you  interact  with  any  other  organizations  or 
individuals  when  performing  your  evaluation 
function? 

d.  What  information  do  you  use  in  your  evaluation 
function? 

e.  What  information  would  you  like  to  have  for 
your  evaluation  function? 

f.  Do  you  have  any  structured  procedures  for  your 
evaluation  function? 

EX:  regulations 

locally  prescribed 
procedures 

g.  What  tools  or  methods  do  you  currently  use  to 
accomplish  your  evaluation  function? 

h.  Do  you  have  access  to  any  tools  or  methods 
which  are  intended  to  help  you  in  your 
evaluation  function  but  which  you  don't  use? 

i.  What  tools  or  methods  would  you  like  to  have  to 
do  your  evaluation  function? 
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j .  Do  you  foresee  any  future  occurrences  or  events 
which  will  impact  training  evaluation? 

k.  Are  there  any  other  problems  in  evaluation  we 
have  not  already  discussed? 

l.  Where  do  you  see  the  highest  payoff  with  regard 
to  the  use  of  tools/methods  in  training 
evaluations? 

m.  Other  persons  who  we  should  talk  to? 

2.  Do  evaluations  impact  the  type  of  training  an  individual 
receives? 

3.  Are  tasks  or  topics  with  greater  mission  importance 
given  greater  weight  in  evaluations? 

4.  What  is  used  to  evaluate  training  requirements  for 
incoming  personnel? 

5.  Do  you  feel  that  evaluations  (i.e.,  knowledge  tests  and 
performance  tests)  fairly  reflect  what  you  have  been 
taught  or  what  you  teach? 

No: 

a.  What  do  you  think  is  wrong  with  current 
evaluations? 

b.  How  should  evaluations  be  conducted? 

6.  Are  you  generally  satisfied  with  training  evaluations? 
E.  Development/Delivery 

"These  remaining  questions  focus  on  the  development  and  delivery  of 
training  within  your  unit.  Development  and  delivery  includes  all 
activities  concerned  with  authoring  and  delivering  instruction  such 
as  training  needs  assessment,  curriculum  development, 
identification  of  delivery  methods,  and  delivering  instruction." 

1.  Are  you  involved  in  the  development  of  training? 

Yes 

a.  Function? 

_  training  development 

method  selection 
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b.  What  specific  products  or  services  do  you 
provide  with  regard  to  your  development/delivery 
function? 


EX:  OPR  for  regulations 

news  letters 
reports 


c.  Do  you  interact  with  any  other  organizations  or 
individuals  when  performing  your 
development /delivery  function? 

d.  What  information  do  you  use  in  your 
development /delivery  function? 

e.  What  information  would  you  like  to  have  for 
your  development /delivery  function? 


f.  Do  you  have  any  structured  procedures  for  your 
development/delivery  function? 

EX:  regulations 

locally  prescribed 
procedures 

g.  What  tools  or  methods  do  you  currently  use  to 
accomplish  your  development /delivery  function? 

develop 

evaluate 

update 

h.  Do  you  have  access  to  any  tools  or  methods 
which  are  intended  to  help  you  in  your 
development/delivery  function  but  which  you 
don't  use? 


i.  What  tools  or  methods  would  you  like  to  have  to 
do  your  development /deli very  function? 


develop 

evaluate 

update 


j.  How  do  you  determine  the  level  to  which  an 
objective  is  taught? 


k.  How  do  you  determine  where  to  teach  a 
particular  task? 
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l.  Do  you  foresee  any  future  occurrences  or  events 
which  will  impact  training  development  and 
delivery? 

m.  Are  there  any  other  problems  in  development  and 
delivery  we  have  not  already  discussed? 

n.  Where  do  you  see  the  highest  payoff  with  regard 
to  the  use  of  tools /methods  in  development  and 
delivery? 

o.  Other  persons  who  we  should  talk  to? 

2.  Do  you  feel  that  you  are  adequately  trained  to  do  your 
job  by  the 

—  unit? 

—  MAT? 

—  FTD? 


a.  How  does  this  impact  your  job? 

b.  Where  should  this  training  occur? 

3.  Are  trainers  in  your  unit  adequately  prepared  to  act  as 
teachers /instructors,  i.e.,  do  they  teach  well? 

—  at  FTDs? 

—  at  MATS? 


No 


a.  Why? 

4.  Do  you  train  anyone? 

Yes: 

a.  What  do  you  train? 

b.  How  are  you  trained  or  qualified  as  a  trainer? 

c.  Do  you  feel  that  you  are  qualified  to  train 
others? 

d.  What  is  your  training  workload? 

e.  Does  this  duty  keep  you  away  from  your  primary 
job? 

f.  How  does  this  impact  your  job  performance? 

g.  Are  additional  personnel  needed? 
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h.  Do  you  feel  that  you  have  adequate  time  to 
train  your  trainees? 

No 

1)  How  does  this  impact  training  quality? 

2 )  How  does  this  impact  the  trainee ' s 
job  performance? 

3 )  How  does  this  impact  unit  performance? 


5.  What  media  are  used  in  your  unit  level  training  programs? 

Lecture /Presentation 
Overhead 
_  CAI 

~  Sound  on  Slide 
_  Video  Tape 

_  IVD 

_____  Text 
__  Hands-on 
~  Other 

6.  On  a  scale  of  1  to  7  with  1  being  very  seldom,  7  being 
very  often,  and  4  being  average,  how  often  is  each  of 
the  media  you  identified  used  in  job  training? 


1 - 2 - 3 - 4 - 5 - 6 - 7 

Very  Avg  Very 

Seldom  Often 

Lecture /Presentation 

Overhead 

CAI 

Sound  on  Slide 
Video  Tape 
IVD 
Text 

Hands-on 

Other 


82 


7.  On  a  scale  of  1  to  7  with  1  being  very  seldom,  7  being 
very  often,  and  4  being  average,  how  often  is  each  of  the 
media  you  identified  used  in  ancillary  training? 

1 - 2 - 3 - 4 - 5 - 6 - 7 

Very  Avg  Very 

Seldom  Often 

Lecture/Presentation 

2 _  Overhead 

_  CAI 

'  Sound  on  Slide 
__  Video  Tape 

_  IVD 

"  Text 

_ _  Hands-on 

__  Other 

8.  Are  the  training  media  you  identified  readily 
accessible? 

No: 

a.  Which  media  are  not  readily  accessible? 

b.  Why? 

9.  How  would  you  assess  the  reliability  of  the  equipment 
used  for  each  of  these  media  on  a  scale  1  to  7  with  1 
being  very  unreliable,  7  being  very  reliable,  and  4 
being  average?  (i.e.,  does  the  equipment  typically  work 
or  not?) 


1 - 2 - 3 - 4 - 5 - 6 - 7 

Very  Avg  Very 

Unreliable  Reliable 

Lecture/Presentation 

Overhead 

CAI 

Sound  on  Slide 

Video  Tape 

IVD 

Text 

Hands-on 

Other 
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10.  How  would  you  assess  the  relative  training  effectiveness 
of  the  training  you  receive  from  each  of  these  media  on 
a  scale  1  to  7  with  1  being  very  ineffective,  7  being 
very  effective,  and  4  being  average? 

1 - 2 - 3 - 4 - 5 - 6 - 7 

Very  Avg  Very 

Ineffective  Effective 

_  Lecture/Presentation 

_  Overhead 

_  CAI 

_  Sound  on  Slide 

_  Video  Tape 

_  IVD 

_  Text 

_  Hands-on 

_  Other 

11.  Do  you  feel  that  your  training  adequately  prepares  you 
for  your  job? 

No: 

a .  Why? 

12.  Do  you  feel  that  the  training  technologies  you  use  today 
adequately  train  what  they  are  intended  to  train? 

No: 

a.  Why? 


13.  Do  you  have  access  to  any  tools  or  methodologies  today 
which  you  don't  use  for  training? 

Yes: 

a.  What  are  they? 

b.  Why  don't  you  use  these  tools  or  methodologies? 


14.  What  tools  or  methods  would  you  like  to  have  for 
training  purposes? 

a.  What  features  are  you  looking  for  in  these 
tools  or  methods? 

b.  Do  you  have  the  resources  to  support  these 
tools  or  methods? 

15.  Where  do  you  see  the  highest  payoff  with  regard  to 
the  use  of  tools  or  methods  in  training? 


8 


16.  Are  you  generally  satisfied  with  the  development  and 
delivery  of  training? 

F.  General  Problems 

1.  What  particular  strengths  and  limitations  in  the  quality 

of  graduates  from  technical  school  have  been  noted  by 
your  unit(s)? 

2.  Do  technical  schools  train  maintainers  in  the  areas  you 
want  them  to? 

3.  Are  technical  schools  overtraining  in  any  areas? 

4.  Are  technical  schools  undertraining  in  any  areas? 

5.  What  is  the  average  time  required  for  an  incoming 
3-level  to  become  position  qualified  in  your  unit(s)? 

—  for  an  incoming  experienced  maintainer? 

6.  How  does  personnel  turnover  in  the  unit  influence 
mission  performance? 

7.  Is  the  level  of  maintainer  experience  declining? 

Yes: 

a.  What  are  the  primary  effects  upon  the 
unit? 

b.  What  are  the  primary  factors  contributing  to 
the  decline? 

c.  What  alternatives  are  available  to  off-set 
these  effects? 


8.  Have  you  encountered  any  other  problems  in  training  that 
we  have  not  already  discussed? 

Yes: 

a.  What  are  they? 

b.  How  do  they  impact  training? 

9.  Are  you  generally  satisfied  with  the  training  process? 

10.  What  changes  should  be  made? 

a.  How  can  tools  or  methods  support  this? 
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11.  Do  you  foresee  any  future  events  or  occurrences  that 
might  impact  the  training  process  in  the  future? 

Yes 

a.  What  are  these  events  or  occurrences? 

b.  What  will  their  impact  be? 

12.  How  do  you  see  training  being  conducted  in 

a.  the  near  term? 

b.  the  long  term? 
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APPENDIX  D:  DATA  ANALYSIS /SUMMARY 


This  Appendix  contains  a  series  of  tables  which  detail  the 
interviewees'  response  to  many  of  the  survey  questions.  For 
each  question,  the  total  number  of  those  respondinq  to  the 
question  is  listed.  Note  that  the  respondent  sample  sizes 
vary  from  question  to  question.  This  was  to  be  expected  since 
not  all  of  the  questions  were  asked  of  each  interviewee  nor 
did  all  of  the  interviewees  give  responses  to  every  question. 
It  should  also  be  pointed  out  that  the  responses  from  each 
group  interview  were  aggregated  and  reduced  to  a  single  data 
point  for  each  response.  In  this  manner,  a  group  of  five  is 
not  weighted  more  heavily  than  a  group  of  three  or  an 
individual  interview.  All  of  the  survey  questions  included  in 
this  Appendix  allowed  for  more  than  one  response.  Thus,  the 
total  number  of  responses  can  exceed  the  sample  size.  The 
frequencies  reported  for  these  items  can  be  interpreted  as 
"the  number  of  individuals  and/or  groups  giving  that 
particular  response."  Percentages  can  be  interpreted 
similarly.  Each  table  contains  a  footnote  reminding  the 
reader  that  respondents  were,  not  restricted  to  one  response. 
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Igfelg  D-l .  Types  of  Training  Requirement  Diagnostics  for  Incoming 
Personnel  (Question  D4);  N  -  75. 


Diagnostic  Cited 


a 

Frequency  Percent 


Initial  interview 

68 

91 

Records  review  (623,  797,  etc.) 

44 

59 

Over-the-shoulder  evaluations 

26 

35 

Computer  RIPs  (CAMS,  CASB,  MMICS,  Personnel) 

16 

21 

Written  test 

2 

3 

Training  rosters 

1 

1 

Peer  group  assessment 

1 

1 

Rivet  Workforce  core  task  list 

1 

1 

a 

Multiple  responses  allowed. 
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Table  D-2.  Decline  of  Maintainer  Experience. 

Factors  Contributing  to  Decline  of  Maintainer  Experience 
(Question  F7);  N  -  50. 

a 

Factor  Cited  Frequency  Percent 


Excessive  turnover 
Ineffective  (or  lack  of)  training 
Rivet  Workforce  policies  have  diluted 
experience . 

Too  much  cross -command /weapon  system  movement 
Early  out  program  has  depleted  experience 
Force  drawdown 

High  year  of  tenure  has  depleted  experience 
levels 

The  Enlisted  Performance  Rating  system  has 
forced  out  good  people. 


29 

58 

15 

30 

15 

30 

15 

30 

6 

12 

6 

12 

6 

12 

2  4 


Multiple  responses  allowed;  60  personnel  responded  that  experience 
was  on  the  decline. 


Alternatives  Available  to  Offset  Decline  in  Maintainer  Experience 
(Question  F7);  N  =  35. 

a 

Alternative  Cited  Frequency  Percent 


Change  the  personnel  policies  24  69 
More  enroute  training  11  31 
Better  training  11  31 


a 

Multiple  responses  allowed. 
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mta. 


labia  D-3 .  Technical  Training  School  C< 

Strangtha  and  Limitation  of  Technical  School  Graduates 
(Question  PI);  N  =  115. 


Strength  or  Limitation  Cited 


a 

Frequency  Percent 


Generally  good  quality  of  graduates 

Some  academic  deficiencies  (AFSC- specific) 

Some  weakness  in  basic  skills/knowledge 

No/limited  contact  with  recent  graduates 

No  trends  have  been  noticed 

Graduates  are  well  disciplined/motivated 

Graduates  lack  military  courtesy/discipline 

Graduates  display  a  negative  attitude 

Graduates  are  confident 

Graduates  display  a  positive  attitude 


42 

37 

31 

27 

20 

17 

18 

16 

11 

10 

9 

8 

5 

4 

3 

3 

1 

1 

1 

1 

a 

Multiple  responses  allowed. 


Areas  of  Overtraining  in  Technical  School  (Question  F3);  N  =  19. 

a 

Response  Frequency  Percent 


Yes,  (AFSC  specific  example)  13  68 

Yes,  the  school  is  too  weapon  system  7  37 

specific 


a 

Multiple  responses  allowed. 
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Table  P-3  ( cont j. 

Areas  of  Undertraining  in  Technical  School  (Question  F4);  N  =  56. 


a 


Type 

of  Training  Cited 

Frequency 

Percent 

Yes, 

Basic  systems  knowledge 

24 

43 

Yes, 

hands-on  skills 

21 

38 

Yes, 

(AFSC- specific  example) 

14 

25 

Yes, 

Technical  Order  use 

8 

14 

Yes, 

troubleshooting 

8 

14 

Yes, 

safety 

2 

4 

Yes, 

maintenance  management  (forms,  MDC, 

etc . )  1 

2 

a 

Multiple  responses  allowed. 
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Table  D-4 .  Types  of  Training  Not  Needed. 

Job/Anclllary  Training  Not  Needed  (Question  A2);  N  =  41. 


Type  of  Training 


a 

Frequency  Percent 


Hangar /Arch  Door  Operation 
Egress 

Block  Training 
Protection  of  the  President 
Aircraft  FOD 

Block  training  (not  relevant  to  AFSC) 

Drug /Alcohol  Abuse 

Ancillary 

Code  of  Conduct 

Flightline  Driving 

Social  Actions 

AGE  Operator 

CPR 

External  Fuel  Tanks 
Fire  Safety 
Standards  of  Conduct 
Tech  School 

ABDR  (Aircraft  Battle  Damage  Repair) 

Aircraft  Corrosion 

Aircraft  Forms 

Block  Training  refresher 

Chemical  Suit 

CUT 

Equipment  Custodian 

FTD  Qualification  (Phased) 

Hazardous  Communication 
Hazards  of  F-16  Aircraft 
Pallet  Buildup  &  Marking 
Self  Aid  &  Buddy  Care 
Task  Qualification  (TQT) 

Tech  Orders /Data 
Job-Related 
Flightline 
M-16 


a 

Multiple  responses  allowed; 
conducted  as  required. 


83  replied  that  training  is  being 
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Safeift  P-4  (cont). 

Reasons  Why  Training  is  Rot  Headed  (Question  A2);  H  =  40. 

a 

Reason  Cited  Frequency  Percent 


Not  relevant  to  AFSC/job  39 
Do  it  frequently  on  the  job  10 
Depends  on  skill  level  or  experience  7 
Quality  of  training  is  not  adequate  6 
Task  too  simple  or  not  critical  5 
Duplicated  in  other  training  programs  3 
Learned  on-the-job  before  attending  class  3 
Never  do  it  on  the  job  2 
Is  contrary  to  what  is  practiced  1 


98 

25 

18 

15 

13 

8 

8 

5 

3 


a 

Multiple  responses  allowed. 
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Table  D-5.  Areas  of  Overtraining. 

Topics  or  Tasks  Overtrained  (Question  A3);  M  =  50. 


Type  of  Training 


a 

Frequency  Percent 


Block  Training 

12 

24 

Chemical  Suit 

10 

20 

Security 

7 

14 

Drug/Alcohol  Abuse 

6 

12 

Egress 

5 

10 

Protection  of  the  President 

4 

8 

Fire  Safety 

3 

6 

Flightline  Driving 

3 

6 

FOD 

3 

6 

Self  Aid  &  Buddy  Care 

3 

6 

AFSC- specific  portions  of  Block  Training 

2 

4 

Aircraft  Corrosion 

2 

4 

Ancillary 

2 

4 

Chafing 

2 

4 

Social  Acti.  is 

2 

4 

Standards  of  Conduct 

2 

4 

AGE  Operator 

1 

2 

CASB 

1 

2 

Code  of  Conduct 

1 

2 

CPR 

1 

2 

External  Fuel  Tanks 

1 

2 

Flightline 

1 

2 

Initial /Qualification 

1 

2 

Integrated  Combat  Turn 

1 

2 

Job-Related 

1 

2 

OJT  Trainer/Supervisor 

1 

2 

Task  Qualification  (TQT) 

1 

2 

Multiple  responses  allowed;  73  replied  that  no  tasks  or  topics  are 
overtrained . 
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Reasons  for  Overtraining  (Question  A3);  H  =  46 


Reason  Cited 


a 

Frequency  Percent 


Training  is  redundant /done  too  frequently 
Do  it  often  enough  to  maintain  proficiency 
Depends  on  skill  level  or  experience 
The  training  is  not  good  (quality  of 
instruction,  realism) 

Task  is  simple  or  not  critical 


35 

76 

19 

41 

12 

26 

7 

15 

2  4 


a 

Multiple  responses  allowed. 
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Tablt  D-6 .  Types  of  Training  Heeded. 

Job/Ancillary  Araas  Hot  Trained,  but  Haadad  (Quant ion  A4);  N  =  53. 

a 

Type  of  Training  Frequency  Percent 


FTD  Qualification  (Phased) 

12 

23 

CAMS 

7 

13 

Enroute 

5 

9 

CASB 

4 

8 

CPR 

4 

8 

Electrostatic  Discharge 

3 

6 

M-16 

3 

6 

AGE  Operator 

2 

4 

Cross  Cultural  Communication 

2 

4 

Hydraulics 

2 

4 

Job-Related 

2 

4 

Refresher  Training  of  infrequent  tasks 

2 

4 

Rivet  Workforce 

2 

4 

Safety 

2 

4 

Aircraft  Familiarization 

1 

2 

Aircraft  Forms 

1 

2 

CTK  Accountability 

1 

2 

Fire  Safety 

1 

2 

Flightline  Driving 

1 

2 

Gun  Boresight 

1 

2 

Hydrazine  Hazard/Response 

1 

2 

Instructor 

1 

2 

Jet  Fuel  Starter 

1 

2 

LOX/LIN  Carts 

1 

2 

Management 

1 

2 

Pallet  Buildup  &  Marking 

1 

2 

Rigging  of  Flight  Control  Systems 

1 

2 

Special  Tools 

1 

2 

Task  Qualification  (TQT) 

1 

2 

Troubleshooting 

1 

2 

Multiple  responses  allowed;  71  replied  that  no  additional  training 
is  needed. 
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Tfrfrlt.  P"6  JS9 afc) . 

Reasons  Training  is  Not  Conducted  (Question  A4);  R  =  46. 


Reason  Cited 


a 

Frequency  Percent 


Training  is  not  offered  39  85 

Qualified  instructor  not  available  15  33 

No  time  available  2  4 

Training  facilities ,  equipment,  materials  2  4 

not  available 


a 

Multiple  responses  allowed. 


* 
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Areas  of  Undertraining. 


Topics  or  Tasks  Undartrainad  (Question  A5);  M  =  64. 


Type  of  Training 


a 

Frequency  Percent 


Self  Aid  &  Buddy  Care 

14 

22 

Chemical  Suit 

13 

20 

CPR 

13 

20 

CAMS 

10 

16 

Troubleshooting 

10 

16 

AGE  Operator 

7 

11 

Disaster  Preparedness 

6 

9 

Fire  Safety 

5 

8 

Bore scope 

4 

6 

Job-Related 

4 

6 

Rivet  Workforce 

3 

5 

Ancillary 

2 

3 

Component  Change 

2 

3 

Drug/Alcohol  Abuse 

2 

3 

OJT  Trainer/Supervisor 

2 

3 

Refresher  Training  of  infrequent  tasks 

2 

3 

Safety 

2 

3 

ABDR  (Aircraft  Battle  Damage  Repair) 

1 

2 

Block  training  (not  relevant  to  AFSC) 

1 

2 

Block  Training 

1 

2 

CUT 

1 

2 

Electrostatic  Discharge 

1 

2 

Enroute 

1 

2 

Flightline  Driving 

1 

2 

FTD  Qualification  (Phased) 

1 

2 

Gun  Bore sight 

1 

2 

Hands-on 

1 

2 

Hangar/Arch  Door  Operation 

1 

2 

Hydrazine  Hazard/Response 

1 

2 

Initial/Qualification 

1 

2 

Instructor 

1 

2 

Integrated  Combat  Turn 

1 

2 

Security 

1 

2 

Social  Actions 

1 

2 

Standards  of  Conduct 

1 

2 

Tech  Orders /Data 

1 

2 

Tech  School 

1 

2 

Multiple  responses  allowed;  58  replied  that  no  topics  or  tasks 
are  undertrained. 


Tmblm  P-7  (oont). 

Reasons  for  Undertraining  (Question  A5);  H  =  49. 


Reason  Cited 


a 

Frequency  Percent 


Task  is  important/critical  37 
Need  hands-on  training  20 
Not  enough  intensity  or  realism  19 
Not  enough  time  17 
Qualified  instructor  not  available  4 
Task  is  complicated/difficult  2 
Facilities,  equipment,  materials  not  available  2 


8 

4 

4 


a 

Multiple  responses  allowed. 
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TabU  P-8.  Typas  of  Feedback  Mechanisms  (Question  B3) ;  N  =  91. 


Feedback  Mechanism 


a 

Frequency  Percent 


Course  critiques 

58 

64 

Informal  coordination  between  training 

43 

47 

agency  and  user 

Training  Evaluation  Reports  (TER) 

25 

27 

Training  Quality  Reports  (TQR) 

22 

24 

Local  questionnaires/surveys 

9 

10 

Training  meetings /conferences 

9 

10 

ECI  Form  17 

6 

7 

AMQP  QA  Reports 

5 

5 

Suggestion  Box 

2 

2 

U  &  T  Workshops 

1 

1 

a 

Multiple  responses  allowed. 
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Tablt  P-9.  Instructor  Qualifications. 


Training/Qualification  to  Serve  as  Instructor  (Question  TB4);  H  = 
55. 

a 

Qualification  Frequency  Percent 


\ 

OJT  Trainer /Supervisor  Course  (FTD)  28  51 

On-the-job  instructor  experience  only  12  22 

Other  military  instructor  training  10  18 

ATC  Technical  Instructor  Course  7  13 

Civilian  education/training/experience  1  2 

Subject  matter  training/certification/  1  2 

experience 

PME  1  2 


a 

Multiple  responses  allowed. 


Training/Qualification  to  Serve  as  Instructor  (Question  EB4): 

OJT  Trainers;  N  =  22 . 

a 

Qualification  Frequency  Percent 


OJT  Trainer /Supervisor  Course  (FTD)  12  55 

On-the-job  instructor  experience  only  8  36 

Other  military  instructor  training  5  23 

ATC  Technical  Instructor  Course  1  5 

Civilian  education/training/experience  1  5 

Subject  matter  training/certification/  1  5 

experience 


a 

Multiple  responses  allowed. 


Table  D-10.  Needed  Training  Technologies. 

Types  of  Training  Tools  or  Methods  Requested  (Question  314);  H  = 
49. 


a 

Response  Frequency  Percent 


IVD 

23 

47 

CAI 

12 

24 

Mock-Up / s imulator 

9 

18 

Video  (local  production  capability) 

7 

14 

Updated  training  materials 

3 

6 

Classrooms 

2 

4 

Yes,  methods  needed  but  not  designated 

2 

4 

System  schematics 

1 

2 

a 

Multiple  responses  allowed. 


Desirable  Features  of  Training  Tools  or  Methods  (Question  E14); 
N  *  31. 


a 

Feature  Cited  Frequency  Percent 


Aid  to  training  difficult  tasks 

15 

48 

( e . g . ,  troubleshooting ) 

Allow  for  hands-on  skill  development 

8 

26 

Easy  to  use 

6 

19 

Flexible 

6 

19 

Convenient 

5 

16 

Realistic 

5 

16 

Self-paced 

4 

13 

Interactive 

4 

13 

Easily  updated 

2 

6 

Standardized 

2 

6 

Understandable 

2 

6 

Minimum  documentation  required 

1 

3 

a 

Multiple  responses  allowed. 
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Tablt  P-11 .  Limiting  Factors. 

Factors  Limiting  Unit  Training  (Question  C2 ) ;  H  =  47 . 

a 

Factor  Cited  Frequency  Percent 


Mission  requirements/operational  commitments  40  85 
Lack  of  emphasis  on  training  10  21 
Availability  of  trainers/trainees  8  17 
Manning  levels  7  15 
Lack  of  qualified  instructors  3  6 
Lack  of  prerequisite  qualification  training  3  6 
Training  is  not  conducted/offered  2  4 


a 

Multiple  responses  allowed. 


Factors  Limiting  FTD  Training  (Question  C2);  N  =  9. 

a 

Factor  Cited  Frequency  Percent 


Lack  of  qualified  instructors  5  56 
Availability  of  trainers /trainees  1  11 
Lack  of  emphasis  on  training  1  11 
Manning  levels  1  11 
Mission  requirements /operational  commitments  1  11 
Lack  of  prerequisite  qualification  training  1  11 
Training  is  not  conducted/offered  1  11 


a 

Multiple  responses  allowed. 

Factors  Limiting  MAT  Training  (Question  C2);  N  =  3. 

a 

Factor  Cited  Frequency  Percent 


Mission  requirements/operational  commitments  2  67 
Manning  levels  1  33 
Training  is  not  conducted/offered  1  33 


a 

Multiple  responses  allowed. 
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Table  D-12.  Effects  of  Personnel  Turnover. 


Effects  of  Personnel  Turnover  on  Mission  Performance  (Question  F6) 
N  =  103. 

a 

Effect  Cited  Frequency  Percent 


It  extends  the  training/qualification  time 
It  impacts  training  (OJT)  continuity 
It  has  little  or  no  influence 
People  PCS  as  soon  as  they  are  trained 
It  has  influence  but  no  reason  specified 
It  influences  people's  attitude 
Since  there  is  no  time  for  training, 
specialize  in  what  they  know 
It  influences  the  mission  most  at  the 
peak  times  (e.g.,  summer,  holidays) 


37 

36 

35 

34 

20 

19 

15 

15 

13 

13 

9 

9 

8 

8 

2 

2 

a 

Multiple  responses  allowed. 


Effects  of  Turnover  on  the  Unit  (Question  F7);  N  =  48. 


Effect  Cited 


a 

Frequency  Percent 


Quality  of  OJT  declines 
Quality  of  maintenance  declines 
More  time  required  for  maintenance 
Morale  suffers 
Increased  training  burden 
Quality  of  Force  suffers 


35 

73 

19 

40 

10 

21 

5 

10 

5 

10 

3 

6 

a 

Multiple  responses  allowed. 


Tabls  D-13.  Training  Transfer  Issues  (Away  frosi  the  Unit). 

Unit  Training  that  Could  be  Conducted  Elsewhere  (Question  A6);  H  = 
38. 


Type  of  Training 


a 

Frequency  Percent 


AGE  Operator 

9 

24 

Hazardous  Communication 

7 

18 

FTD  Qualification  (Phased) 

6 

16 

Flightline  Driving 

5 

13 

Rivet  Workforce 

5 

13 

Self  Aid  &  Buddy  Care 

5 

13 

CPR 

3 

13 

Fire  Safety 

2 

5 

Ancillary 

1 

3 

Chemical  Suit 

1 

3 

Component  Change 

1 

3 

Conversion 

1 

3 

Egreos 

1 

3 

Electrostatic  Discharge 

1 

3 

External  Fuel  Tanks 

1 

3 

M-16 

1 

3 

Refresher  Training  of  infrequent  tasks 

1 

3 

Security 

1 

3 

Task  Qualification  (TQT) 

1 

3 

Tech  Orders /Data 

1 

3 

Towing 

1 

3 

Troubleshooting 

l 

3 

a 

Multiple  responses  allowed. 


Sites  for  Training  (Question  A6);  H  =  38. 

a 

Location  Cited  Frequency  Percent 


FTD 

15 

39 

MAT 

14 

37 

Tech  School 

12 

32 

Wing 

6 

16 

AGE  Shop 

5 

13 

Hospital 

4 

11 

Enroute 

3 

8 

105 


a 

Multiple  responses  allowed. 

Table  D-13  (cont). 

Reasons  for  Transferring  Training  (Question  A6);  N  -  22. 

a 

Reason  Cited  Frequency  Percent 


Qualified  instructor  not  available  in  unit  22  100 
Need  to  be  able  to  do  immediately  on  the  job  6  27 
Training  needs  to  be  standardized  6  27 
Don't  have  time  4  18 
Important /complex  enough  for  formal  training  4  18 
Work  environment  not  conducive  to  training  3  14 
Facilities,  equipment,  materials  not  available  1  5 


a 

Multiple  responses  allowed. 


106 


r. 


Table  P-14.  Training  Transfer  issues  (To  the  Unit). 

Training  that  Should  be  Conducted  at  the  Unit  (Question  A7);  M  = 
48. 


Type  of  Training 


a 

Frequency  Percent 


Hangar /Arch  Door  Operation 

9 

19 

Fire  Safety 

9 

19 

CAMS 

8 

17 

CPR 

8 

17 

Self  Aid  &  Buddy  Care 

6 

13 

Block  Training 

5 

10 

Egress 

5 

10 

Chemical  Suit 

4 

8 

AGE  Operator 

3 

6 

Chafing 

3 

6 

Towing 

3 

6 

AFSC-specific  portions  of  Block  Training 

2 

4 

FOD 

2 

4 

CASB 

2 

4 

Hazardous  Communication 

2 

4 

Tech  School 

2 

4 

Code  of  Conduct 

1 

2 

Electrostatic  Discharge 

1 

2 

Initial /Qualification 

1 

2 

Safety 

1 

2 

Standards  of  Conduct 

1 

2 

a 

Multiple  responses  allowed. 
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Table  D-14  (crot|. 

Reasons  for  Transferring  Training  to  the  Unit  (Question  A7);  M  = 
44. 


a 

Reason  Cited  Frequency  Percent 


Training  should  to  be  tailored  to  unit  needs  37 
Unit  personnel  are  most  qualified  to  teach  it  12 
Allows  flexibility  in  scheduling  training  8 
Lose  people  from  work  longer  than  necessary  7 
Ease  of  administration  6 
Task  difficulty  does  not  justify  formal  class  5 
Training  needs  to  be  hands-on  3 


84 

27 

18 

16 

14 

11 

7 


a 

Multiple  responses  allowed. 
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Table  P-15.  Training  Importance  Ratings 


1/3  Level _ 5  Level _ 7  Level 


Type  of 


Training 

Mean 

SD 

N 

Mean 

SD 

N 

Mean 

SD 

N 

ANCILLARY 

Chem  Suit 

4.35 

.91 

209 

4.21 

.91 

218 

4.03 

1.13 

201 

Fire  Safety 

4.47 

.75 

208 

4.22 

.88 

216 

4.06 

1.10 

199 

Age  Operator 

3.60 

1.23 

204 

3.57 

1.09 

211 

3.26 

1.29 

194 

Block  Trng 

3.96 

1.19 

207 

3.47 

1.18 

215 

3.20 

1.31 

197 

Hangar  Door 

3.02 

1.36 

203 

2.96 

1.26 

211 

2.80 

1.32 

194 

Operation 

CPR 

4.15 

1.02 

209 

4.18 

.99 

216 

4.17 

1.00 

198 

Self -Aid/ 
Buddy  Care 
Security, 

4.30 

.91 

211 

4.18 

.93 

218 

4.13 

1.02 

200 

4.35 

.91 

211 

4.20 

.89 

218 

4.16 

1.04 

200 

COMSEC, 

OPSEC 

Flightline 

3.46 

1.39 

209 

3.73 

.99 

217 

3.70 

1.12 

199 

Driving 

Drug/Alcohol 

4.13 

1.02 

210 

3.82 

1.10 

217 

3.71 

1.21 

199 

Abuse 

Soc  Actions 

3.67 

1.12 

209 

3.46 

1.09 

216 

3.45 

1.20 

198 

Disaster 

3.99 

1.04 

210 

3.89 

.98 

217 

3.89 

1.12 

199 

Prep. 

Standards  of 

4.14 

.96 

210 

3.84 

1.02 

217 

3.78 

1.21 

199 

Conduct 

Protection 

3.63 

1.34 

208 

3.54 

1.34 

216 

3.49 

1.34 

198 

of  President 

Code  of 

3.91 

1.14 

210 

3.71 

1.15 

217 

3.66 

1.22 

199 

Conduct 
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Training  Importance  Ratings 


1/3  Level _ 5  Level _ 7  Level 

Type  of 


Training 

Mean 

SD 

N 

Mean 

SD 

N 

Mean 

SD 

N 

JOB-RELATED 

• 

Initial/Qual 

4.53 

.79 

202 

4.14 

.96 

206 

3.81 

1.20 

189 

FTD  (AMQP) 

4.07 

1.17 

192 

3.88 

1.05 

200 

3.55 

1.16 

182 

Continuation 

3.91 

1.10 

198 

3.88 

.93 

206 

3.68 

1.12 

189 

Requal 

3.52 

1.27 

193 

3.76 

1.04 

204 

3.73 

1.09 

187 

Conversion 

3.34 

1.42 

187 

3.69 

1.14 

196 

3.71 

1.19 

180 

CUT 

2.60 

1.39 

182 

3.14 

1.26 

191 

3.21 

1.29 

175 

Note.  Ratings  were  on  a  scale  of  1  (Not  Important)  to  5  (Extremely 
Important ) .  Rating  form  ( See  Appendix  C )  was  administered 
individually  to  each  interviewee. 
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APPENDIX  E:  HSD  TECHNOLOGIES 


Current  HSD  Science 

Advanced  Instructional  Design  Advisor  (AIDA1  research  is  exploring 
the  automation  of  an  instructional  design  system  which  would  guide 
course  developers  through  the  instructional  design  process.  The 
system  would  be  derived  from  empirically  sound  and  theory-based 
instructional  design  model. 

Adyanced-On-the- Job  Training  System  l AOTS 1  will  integrate  training 
development,  delivery/  evaluation,  and  management  to  provide  an 
automated  squadron-level  technology  for  systematic  and  effective 
job-site  training.  The  production  version  of  this  technology,  the 
Base  Training  System  (BTS),  will  be  limited  to  the  management 
functions  of  AOTS  prototype. 

Advanced  Training  System  f ATS )  is  an  integrated  CBT  and  support 
system  using  distributed  architecture  for  development,  delivery, 
evaluation,  and  management  of  training  at  ATC  resident  technical 
training  wings.  Instructional  development  subsystems  could  be  used 
for  contractor-produced  training  systems  which  would  then  be 
exported  to  units.  ATS  has  the  potential  to  be  the  foundation  for 
all  CBT,  simulators,  and  part-task  trainers.  Instructional 
development,  delivery,  management,  and  evaluation  subsystems 
developed  under  this  effort  could  be  applied  to  SLT. 

Aircrew  Combat  Mission  Enhancement  f ACME )  will  provide  a  low-cost 
method  for  defining  multiship  team  training  issues  and  problems. 
Remote  Air  Intercept  Trainers  could  be  networked  to  ACME  for  SLT. 
Basic  Job  Skills  (BJS)  is  a  type  of  intelligent  tutor  that  can  be 
used  for  training  first  term  airman  in  high  tech  jobs.  The  goal  is 
to  train  technicians  who  can  adapt  across  generations  of  weapon 
systems  and  AFSCs.  The  training  emphasizes  general  technical 
knowledge  and  skill  and  the  importance  of  performance  with 
understanding . 

Basic  Job  Skills  project  is  developing  an  integrated  cognitive  task 
analysis/training  development  technology  to  build  adaptive  training 
directed  at  the  fundamental  mental  skills  required  in  the  AF's  most 
technologically  advanced  enlisted  occupations.  The  goal  is  to 
train  the  complex  problem  solving  skills  that  accelerate  the  growth 
to  competence  and  to  knowledge  flexibility  in  high-tech  workcenters 
by  capturing  the  collective  maintenance  savvy  of  expert 
technicians.  The  expert  knowledge  base  is  then  transformed  into 
learnable  curriculum  for  novice  technicians. 

C-130  Aircrew  Training  System  Evaluation  (ATSE1  will  produce  system 
evaluation  methodology  to  help  determine  if  future  simulators  are 
effective  training  devices.  The  methodology  will  measure  pilot 
proficiency  effected  by  the  simulator. 
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Comprehensive  Occupational  Data  Analysis  Program  (CODAP)  is  a 
package  of  computer  programs  used  to  input,  process,  organize,  and 
report  occupational  data  from  job  inventories.  CODAP  manipulates 
and  reports  task-level  and  biographical  survey  data  gathered  from 
job  incumbents  and  expert  raters  for  the  purpose  of  identifying  and 
analyzing  current  job  structures  and  task  characteristics  within  a 
target  occupational  area. 

Computer-Based  Training  Selection  and  Implementation  Strategies 
will  explore,  evaluate,  and  develop  guidelines  for  selecting  and 
applying  CBT  technologies  to  training.  The  products  will  guide 
training  managers  in  selecting  and  converting  existing  courses  into 
CBT  systems. 

Fundamental  Skills  Training  (FST)  research  involves  an  assessment 
of  those  fundamental  skills  that  airmen  or  recruits  must  possess  if 
they  are  to  succeed  in  the  AF. 

Instructional  Support  System  f ISS )  provides  a  government-owned 
software  package  for  use  in  computer-based  training  and  management. 
A  user-friendly  system  hosted  on  a  micro-computer,  ISS  would  allow 
trainers  at  the  squadron- level  to  develop  CBT  for  their  own  use 
which  can  be  transported  to  other  units. 

Integrated  Maintenance  Information  System  fIMISl  can  be  used  to 
provide  in-depth  training  for  the  maintenance  technician  through  a 
portable  digital  text  and  graphics  display  job  aid.  This 
electronic  job  aid  can  be  hand-carried  to  present  the  required 
information  at  the  job  site. 

Intelligent  Tutoring  Systems  (ITS)  will  provide  effective,  low 
cost,  transportable  training  for  individualized  instruction  in 
specific  high  technology  areas.  ITS  authoring  and  delivery 
capabilities  will  result  from  the  Intelligent  Computer  Assisted 
Training  Testbeds  project. 

Job-Aided/Training  Analysis  Technologies  ( JATAT 1  will  provide 
methods  for  deciding  whether  and  how  tasks  should  be  trained, 
job-aided,  or  some  combination  of  the  two  and  provide  human  and 
system  performance  models  to  evaluate  those  decisions. 

Job  Performance  Measurement  l  JPM)  program  is  developing  methods  for 
measuring  individual  job  performance  to  validate 
selection/classification  tests,  evaluate  training  systems /programs, 
and  evaluate  research  products  (e.g.,  tutors). 

Logistics  Command  and  Control  focuses  on  the  development  of 
empirically-based  integrated  instructional  methodologies  for 
individualized  instruction  within  the  logistics  command  and  control 
environment.  Specifically,  integrated  approaches  to  provide  self- 
paced  instruction  of  prerequisite  knowledges  and  skills, 
orientation  and  preliminary  indoctrination  to  individual  roles,  and 
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maintenance  of  proficiency  for  LC2  battle  staffs.  The  LC2  program 
will  develop  an  integrated  instructional  methodology  and  a 
prototype  desktop  trainer  for  the  logistics  battle  staff 
environment . 

Multi-Task  Trainer  Research  (MTTR1  is  based  on  Air  Intercept 
Trainer  technology  and  will  provide  emergency  procedure  training  in 
flight  safety.  It  will  have  authoring  capabilities  so  that 
non-programmers  can  quickly  and  easily  modify  course  content  to 
provide  up-to-date  training. 

Part  Task  Trainers  (PTTs )  provide  low  cost,  off-the-shelf,  easily 
upgradeable  simulator  training  systems.  PTTs  would  permit  frequent 
practice  of  mission  critical  tasks  at  the  squadron  level. 

Task  Analysis  for  Tactical  C2  (TATC2J  will  provide  knowledge 
engineering  methodologies  for  command  and  control  domains.  This 
will  alleviate  some  of  the  problems  associated  with  developing  new 
training  systems. 

Training  Assessment  Technologies  can  provide  methods  for  developing 
measures  of  job  knowledge  and  performance  which  can  be  used  to 
identify  training  needs  and  evaluate  training  programs  and 
individual '8  performance.  One  example  is  the  evaluation  of  ATC/TAC 
4 -level  training  program. 

Training  Decision  Modeling  Technologies  fTDMTl  will  allow  training 
managers  to  make  better  decisions  about  the  what,  when,  and  where 
of  training.  Decisions  could  be  made  that  focus  on  the 
squadron- level  training  needs. 

Training  System  for  Maintenance  (TRANSFORM)  will  be  used  by 
Instructional  Systems  Developers  in  developing  maintenance  training 
for  new  weapon  systems.  The  TRANSFORM  products  will  be  used  by 
Field  Training  Detachments.  The  system,  which  provides  an 
automated  interface  between  the  Logistics  Support  Analysis  process 
and  Instructional  Systems  Development,  is  being  used  as  a  prototype 
in  a  Joint  Service  Decision  Support  System. 
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Related  BSD  Technologies 


3-Dimensional  Audio  research  explores  technology  that  can  project 
sound  in  a  three  dimensional  space  using  a  headset  and  the  implica¬ 
tions  of  human  abilities  to  localize  that  information.  With  future 
development,  this  technology  could  be  applied  as  an  output 
technique  in  computer-based  training  systems  deployed  at  the  unit 
level.  (AAMRL/BB) 

Air  Crew  Selection  research  continues  to  develop  high  performance 
skill  assessment  techniques  that  potentially  could  be  applied  to 
the  assessment  of  an  individual ' s  skill  level  as  the  student 
progresses  through  high  performance  training.  (AL/HRMO) 

Aircraft  Mishap  Prevention  related  research  is  developing  an  inte¬ 
grated  database  and  analysis  system  for  processing  mishap-related 
information.  This  system  can  be  used  for  rapid  dissemination  of 
mishap  prevention  information  to  networked  units.  (HSD/YA) 

B-1B  Engineering  Research  Simulators  efforts  will  derive  human 
factors  principles  on  human-machine  interactions  which  may  have 
implications  for  simulators  and  other  computerized  training 
systems.  (AAMRL/HE ) 

Cockpit  Automation  Technology  (CAT)  can  contribute  to  SLT  by 
providing  engineering  design  tools  which  would  be  used  in  the  rapid 
prototyping  of  high  fidelity  simulations.  Research  on  simulation 
validation  techniques  may  also  contribute  to  the  development, 
delivery,  and  evaluation  of  SLT  systems.  (HSD/YA) 

Command.  Control-  Communication  (C3>  research  can  facilitate  the 
development  of  realistic  mission  environment  simulations.  These 
simulations  can  be  used  to  train  team  decision-making  and  other 
team  related  activities.  Human-computer  interaction  in 

computer-based  training  systems  may  be  improved  by  the  research 
efforts  in  C3.  (AAMRL/HE,  AL/HRL) 

Duodecahedron  Screens  with  back-project  video  around  a  cockpit 
development  offers  relatively  low-cost  flying  simulations.  This 
technology  has  the  potential  to  be  deployed  at  unit  training 
centers.  (AL/HRA) 

Human  Performance  in  Simulations  research  studies  various  effects 
on  human  performance  in  simulators.  Investigations  include 
exploring  transfer  of  training  across  simulators,  effects  of 
simulator  performance  (e.g.,  timing  delays)  on  human  performance, 
and  the  role  of  visual  cues  in  low-level  flying.  (AAMRL/HE) 
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Integrated  Perceptual  Information  for  Designers  (IPID)  efforts 
produced  reference  materials  on  human  performance  specifications 
relevant  to  system  design.  These  materials  have  implications  for 
simulation  development.  (AAMRL/HE) 

Learning  Abilities  Measurement  Program  t LAMP )  conducts  research  on 
human  learning,  memory,  and  problem  solving  abilities.  The 
outcomes  of  this  research  provides  data,  principles,  and  theory 
concerning  human  learning  that  can  be  embedded  in  SLT  systems. 
(AL/HRMO) 

Low  Fidelity  Helmet  development  efforts  provide  a  low-cost 
alternative  to  high-cost  fiber  optic  helmets.  These  developments 
can  be  applied  to  virtual  world  applications,  simulators,  and 
computer-based  training  systems.  (AL/HRA) 

Low  Level  Flying  Trainer  efforts  have  applied  videodisc  technology 
to  teaching  principles  of  low  level  flying  training.  This 
relatively  inexpensive  technology  could  be  adapted  for  SLT. 
(AL/HRA) 

Operator  Workload  Measurement  research  is  designed  toward 
developing  techniques  for  measuring  operator  mental  workload  in 
complex  tasks.  These  measurement  techniques  may  be  applied  in  the 
development,  delivery,  and  evaluation  of  SLT  systems.  (AAMRL/HE) 

Speaker  Independent.  Continuous  Speech  Recognition  is  a  problem 
that  plagues  systems  requiring  natural  human-to-machine 
interaction.  Efforts  within  HSD  are  attempting  to  develop  systems 
that  recognize  human  speech  without  "training"  that  system  first. 
Computer-based  training  systems  will  greatly  benefit  from  advances 
in  this  field.  (AAMRL/BB) 

Super  Cockpit  research  has  the  potential  to  enhance  SLT  delivery 
through  its  research  on  crew-cockpit  interface  technology  (i.e., 
man-machine  interface).  This  technology  can  be  integrated  into 
visual  display  devices  to  enhance  flight  training.  (HSD/YA) 

Virtual  Man-Machine  Interaction  related  projects  are  being 
conducted  to  explore  virtual  space  simulators.  Low-cost  virtual 
worlds  can  be  applied  to  training  novices  to  fix  technologically 
complex  equipment  in  a  virtual  environment.  This  results  in 
greater  opportunities  for  training  in  "safe"  environments. 
(AAMRL/HE) 
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HSD  Specific  Training  Domains 


In  addition  to  conducting  research  relevant  to  SLT,  HSD  has  unique 
expertise  that  can  be  tapped  in  the  development  of  specific  unit 
level  courses.  These  courses  include: 

Aerospace  Medical  Education 
Hazardous  Material  Management 
Physiological  Training 
Occupational  and  Environmental  Health 
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APPENDIX  F:  DEFINITIONS  OF  ISD  PROCESSES 


ANALYZE  SYSTEM  REQUIREMENTS 

1.  Training  Needs  Assessment 

a.  Analyze  Situation.  Determine  whether  or  not  a 
training/education  option  is  appropriate. 

b.  Identify  Parameters.  Identify  operational 
requirements  and  constraints,  design  drivers,  and 
system  selection  criteria. 

2.  Define/Analyze  Job  Performance  Requirements 

a.  Develop  Task  Listing.  Describe  what  people  must  do 
to  perform  their  jobs  in  terms  of  tasks,  subtasks  and 
activities . 

b.  Analyze  Job  Tasks.  Identify  the  skills,  knowledges, 
proficiency  levels,  and  supporting  competencies 
required  by  the  target  population  for  successful 
accomplishment  of  the  job. 

c.  Estimate  Target  Population  Characteristics.  Survey 
or  estimate  the  skills,  knowledges,  and  supporting 
competencies  of  the  target  population. 

DEFINE  EDUCATIONAL  TRAINING  REQUIREMENTS 

1.  Select  Job  Tasks  Requiring  Training.  Examine  each  task  to 
determine  if  it  requires  training. 

2.  Determine  Student  Prerequisites.  Determine  the 
prerequisite  skills,  knowledges  and  supporting 
competencies  required  of  individuals  entering  training  for 
each  task. 

3.  Select  Appropriate  Training  Setting.  Determine  the 
appropriate  type  of  training  setting  (i.e.,  resident  course, 
field  training,  OJT,  mobile  training  team,  correspondence, 
job  aid) . 

4.  Estimate  Resource/Logistic  Requirements.  Estimate 
facilities,  equipment,  funding  (including  development  and 
O&M) ,  and  personnel  requirements. 
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DEVELOP  OBJECTIVES  AND  TESTS 


1.  Develop  Objectives.  Describe  observable  behavior,  minimum 
standards  of  performance,  or  proficiency  expected,  and 

the  conditions  under  which  the  behavior  is  to  be  exhibited. 

2.  Develop  Tests.  Develop  tests  to  measure  achievement  under 
the  conditions  and  standards  specified  by  the  objectives. 

3.  Develop  Job  Performance  Testing  Standards.  Describe 
the  procedures  to  be  used  by  the  evaluators  when 
evaluating  job  performance. 

PLAN,  DEVELOP  &  VALIDATE  INSTRUCTION 

1.  Plan  Sequence  of  Instruction.  Arrange  sequence  of  learning 
objectives  so  that  prior  learning  becomes  the  context 

or  acquisition  of  new  knowledge. 

2 .  Select  Instructional  Method 

a.  Evaluate  Alternative  Instructional  Methods.  Evaluate 
alternative  course  design  stratagies,  teaching  models  and 
methods,  and  evaluation  strategies  against  the  type,  kind 
and  level  of  training  and  student  performance  anticipated 

b.  Identify  Instructional  Method.  Identify  the  way  by  which 
instruction  will  be  designed,  structured,  and  evaluated. 

c.  Establish  Detailed  Course  Design.  Organize 
instructional  events  (e.g.,  gaining  attention,  informing 
learner  of  the  objective,  stimulating  recall  of  prior 
knowledge,  presenting  stimulus  material,  providing 
feedback,  etc.)  to  support  learning  processes. 

2.  Select  Instructional  Media 

a.  Evaluate  Candidate  Media.  Survey  capabilities  of 
candidate  training  technologies  or  media  and  compare 
to  system  criteria. 

b.  Select  Instructional  Media.  Identify  the  means  by  which 
the  instruction  will  be  delivered  to  learners  (e.g., 
interactive  video,  computer,  lecture,  books,  etc.) 

c.  Develop  system  specification.  Define  and  analyze  a 
complete  set  of  functional,  interface,  and  performance 
requirements  for  the  instructional  system. 

3.  Determine  Resource  and  Funding  Requirements.  Determine 
facility,  equipment,  personnel  and  funding  requirements  for 
instruction. 


4. 


Develop  Instructional  Materials 


a.  Author  Instructional  Materials.  Prepare  lesson 
specifications ,  script  lessons,  layout  Storyboards  for 
each  lesson  and  test,  and  edit  scripts,  storyboards, 
lesson  flow  diagrams,  etc. 

b.  Produce  Instructional  Materials.  Develop  instructional 
products,  such  as  videos,  computer  programs,  and 
simulations,  based  on  lesson  specifications 

5.  Validate  Instructional  Materials 

a.  Review  Courseware  Prototype.  SME  and  ISD  review  of 
courseware  prototypes,  lessons  and  test  specifications, 
storyboards,  layouts,  art  etc. 

b.  Individual  student  tryouts  of  instruction.  Pretest 
instructional  material  on  selected  individuals. 

c.  Small  group  tryouts  of  instruction.  Pretest 
instructional  material  on  selected  small  groups. 

6.  Validate  Complete  System.  Try  out  instructional  material  on 
a  sample  of  students  who  are  representative  of  the  total 
target  population. 

E.  CONDUCT  AND  EVALUATE 

1.  Deliver  Instruction.  Present  instructional  materials 

according  to  the  strategy  and  media  selected. 

2.  Support  Instruction 

a.  Schedule  students.  Ensure  students  are  scheduled  for 
required  training  in  a  timely  manner. 

b.  Schedule  resources.  Provide  for  resources  to  support 
operation  of  the  instructional  system,  including 
equipment,  lesson  materials,  instructors  and 
supervisors. 

c.  Track  student  progress.  Monitor  students  to 
determine  where  they  are  in  their  training  program, 
where  they  should  be,  and  what  has  to  be  done  to  get 
them  there. 

d.  Manage  training  requirements.  Ensure  that  training 
requirements  reflect  those  tasks,  skills  and 
knowledges  which  people  must  know  to  do  their  jobs 
and  which  require  training. 
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e.  Maintain  and  update  instructional  system.  Provide 
for  maintenance  and  currency  of  instructional  system. 

3 .  Evaluate  Instruction 

a.  Evaluate  internal  validity  of  instructional  product 
(student  performance).  Assess  effectiveness  of  an 
instructional  program  in  terms  of  student  performance. 

b.  Evaluate  external  validity  of  instructional  product 
(job  performance).  Assess  effectiveness  of  an 
instructional  program  in  terms  of  performance  on  the 
job. 

c.  Measure  training  system  utility  and  cost-benefit. 

Assess  the  effectiveness  of  the  instructional  program  in 
light  of  its  cost. 
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APPENDIX  G:  CRITERIA  FOR  EVALUATION  OF  TRAINING  TECHNOLOGIES 


A  key  component  of  the  SLT  research  initiative  is  the 
assessment  of  HSD  training  technologies  for  their  potential 
usefulness  at  the  unit  level.  These  technologies  are  to  be 
assessed  at  some  future  point  to  allow  Air  Force  decision-makers  to 
prioritize  R&D  support  for  both  existing  and  currently 
non-existent,  but  needed,  technologies.  The  establishment  of 
evaluative  criteria  is  a  preliminary  stage  required  for  advancement 
of  a  Science  and  Technology  (S&T)  Investment  Strategy  which  will 
prescribe  R&D  activity  in  the  area  of  training  technology 
development.  This  Appendix  will  describe  the  identification  and 
development  of  a  set  of  proposed  evaluative  criteria  and  present 
strategies  and  sample  instruments  for  gathering  the  data  necessary 
to  allow  evaluation  and  prioritization  of  technologies.  Finally, 
this  appendix  examines  some  potential  benefits  associated  with 
enhanced  SLT  based  on  project  manager  inputs. 

A.  The  Need  For  Criterion  Development 

In  the  current  environment  of  constrained  resources  (i.e., 
funding,  staffing,  time),  researchers  and  developers  will  likely  be 
increasingly  limited  in  the  number  of  technologies  that  they  can 
pursue  in  the  development  of  new  training  systems.  Therefore,  it 
is  of  extreme  importance  that  the  "right”  technologies  be 
identified  as  early  as  possible  in  the  development  process. 
Important,  too,  is  that  efforts  and  resources  be  concentrated  to 
apply  the  selected  technology  to  the  training  need  rapidly  and 
economically. 

To  effectively  operate  in  this  environment,  developers  must 
have  an  effective  set  of  standards,  measures,  or  gauges  by  which 
they  can  evaluate  the  potential  of  a  training  technology.  These 
measures  become  the  "criteria"  for  evaluating  the  utility  of  the 
application  of  technologies  to  the  various  training  needs.  The  use 
of  the  term  "utility"  has  a  number  of  connotations  for  researchers 
and  evaluators.  For  this  study,  "utility"  can  be  thought  of  as 
advantage,  applicability,  appropriateness,  fitness,  relevance,  or 
usefulness.  That  is,  the  focus  of  evaluation  is  to  assess  the 
capability  of  or  potential  for  technologies  to  meet  identified 
training  needs. 

While  little  has  been  written  by  researchers  on  the 
development  of  criteria  to  be  used  for  evaluation,  a  brief 
discussion  of  performance  criterion  development  can  be  helpful 
(e.g.,  tests,  ratings).  With  this  related  literature  as  a  basis 
for  the  present  effort  (cf..  Smith,  1976),  guidelines  for  the 
evaluative  criteria  themselves  can  be  extracted.  For  example. 
Bellows  (1961)  indicated  that  measures  should  be  reliable, 
realistic,  representative,  predictable,  relevant,  acceptable  to 
users,  and  consistent  from  one  situation  to  another.  Blum  and 
Naylor  (1968)  specified  that  criterion  measures  should  also  be 
inexpensive,  understandable,  measurable,  relevant,  uncontaminated 
and  bias-free,  and  discriminating.  Thus,  we  have  some  general 
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guidance  and  standards  for  the  identification  of  criteria  to  be 
used  for  the  evaluation  of  training  technologies. 

The  following  discussion  identifies  several  criteria  that  have 
the  potential  for  identification  of  those  candidate  technologies 
with  the  greatest  potential  of  providing  the  desired  training 
solutions  (i.e.,  increasing  training  effectiveness  and  efficiency). 
However,  it  must  be  realized  that  these  criteria  cannot  be 
identified  and  applied  without  parallel  development  of  measurement 
procedures  to  ensure  that  diverse,  valid  organizational  interests 
are  properly  identified  and  included  in  the  evaluation  process.  As 
an  example,  field  comments  from  the  SLT  interview  process  revealed 
that  unit-level  trainers  are  most  interested  in  the  development  and 
delivery  of  new  hands-on  training  technologies  which  supplement  and 
enhance  OJT.  They  are  not  concerned  about  state-of-the-art 
educational  theories  or  technologies.  Nor  are  they  concerned  with 
the  planning/programming  and  management /evaluation  domains  of 
higher  headquarters.  They  are  concerned,  however,  with  some 
management  aspects  such  as  record  keeping,  and  want  an  input  on 
determining  what  to  train,  when  to  train,  and  how  best  to  train. 

Thus,  evaluative  criteria  and  the  associated  measurement  and 
weighting  procedures  must  not  only  properly  identify  and  reflect 
these  unit-level  concerns,  but  must  do  the  same  for  the 
intermediate  managers  and  planners,  headquarters  staff,  and  other 
concerned  parties.  The  discussion  of  each  criterion  below  will 
include  suggestions  for  quantifying  each  criterion  and  for 
gathering  data  from  the  appropriate  sources  across  organizational 
levels. 


B.  SLT  Evaluative  Criteria 

Many  evaluative  criteria  relevant  to  this  project  are 
conceptually  related  and  interdependent  (e.g.,  "efficiency"  and 
"cost";  "acceptable"  and  "user-friendly").  For  the  purposes  of 
selection  of  criteria  here,  however,  it  is  important  that  each 
criterion  be  distinct  from  all  others  to  allow  for  greater 
precision  in  analysis  and  interpretation  of  results.  It  is 
necessary  that  each  criterion  be  clearly  defined  and 
operationalized  for  usage  as  intended  by:  (a)  those  subject-matter 
experts  (SMEs)  supplying  criterion  evaluation  data;  (b)  researchers 
and  decision-makers  interpreting  SME  responses;  and  (c)  researchers 
and  decision-makers  evaluating  existing  sources  of  technical  data 
and  other  input  (e.g.,  test  results,  records,  evaluative  study 
findings).  The  following  section  presents  and  defines  each 

evaluative  criterion,  provides  discussion  of  the  rationale  for 
inclusion,  and  proposes  measurement  tactics  for  collecting  the 
necessary  data. 

The  individual  evaluative  criteria  are  grouped  under  broad 
categories  of  either  "Unit-Level  User  Acceptability"  or  "Management 
Acceptability."  We  believe  that  this  general  sorting  of  criteria 
helps  to  reflect  the  necessary  duality  of  evaluative  focus  required 
for  most  R&D  efforts  and  accurately  depicts  the 

multi-dimensionality  of  the  "Acceptability"  criteria  (i.e.,  there 
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are  many  contributors  to  the  perception  of  acceptability ) .  This 
two-sided  view  can  be  represented  by  the  extreme  case  where 
"management"  forces  new  procedures,  gadgets,  documentation,  among 
others,  on  the  work  force  at  the  unit  level  (i.e.,  the  "users") 
without  fully  considering  the  necessity  of  the  new  technology  from 
the  users'  point.  Given  that  the  two  viewpoints  have  different 
concerns,  it  is  essential  that  both  be  included  in  any  evaluation 
of  technologies  designated  for  operational  implementation.  Some  of 
the  criteria  listed  below  are  relevant  to  only  one  source  (i.e., 
user  or  management ) ,  whereas  others  are  proposed  for  multiple 
sources  to  gather  the  widest  possible  range  of  valid  input. 

It  is  important  to  note  that  the  current  developmental  stage 
of  the  various  technologies  to  be  considered  will  differ, 
reflecting  their  states  of  advancement.  Thus,  some  of  the  data 
gathered  will  be  somewhat  speculative  in  nature,  relying  on 
incomplete  data,  reports,  preliminary  test  findings,  and  so  forth. 
Criteria  requiring  expert  and/or  user  ratings  will  be  based  on 
judgments  of  prototype  features,  projected  requirements,  and 
planned  procedures.  The  following  criteria  can  be  applied  to  new 
technologies  across  the  range  of  developmental  phases  even  though 
the  type  and,  perhaps,  validity  of  input  data  may  vary. 

Unit-Level  User  Acceptability 

Criteria  relevant  to  this  concern  reflect  the  likelihood  that: 
(a)  the  unit  will  use  the  technology,  (b)  the  unit  will  use  the 
technology  correctly,  and  (c)  the  users  will  see  the  technology  as 
an  asset  with  value  for  training  at  the  unit  level.  Developers  who 
do  not  take  these  criteria  into  consideration  will  likely  fail  to 
provide  the  user  community  with  an  effective  technology. 

Useability.  This  criterion  reflects  the  convenience  and  ease 
of  use  of  the  technology.  Included  in  this  consideration  would  be 
the  users'  comprehension  of  instructions  (i.e.,  understandability ) . 
There  should  be  an  absolute  minimum  of  training  and  reading  of 
instructions  required  to  be  able  to  use  the  technology,  especially 
for  those  whose  primary  job  is  not  training.  Data  should  be 
collected  from  the  users,  including  trainees  and  trainers,  in  the 
form  of  ratings  of  convenience  and  understandability.  An  analysis 
of  reading  level  difficulty  is  an  appropriate  measure  to  be 
considered  in  the  determination  of  useability.  Additionally,  an 
assessment  measure  or  "test"  of  the  comprehension  of  instructions 
and  use  of  the  technology  could  be  developed  and  administered. 
This  criterion  is  particularly  important  when  the  technology  is 
hardware-oriented,  involving  sophisticated  and/or  novel  equipment. 

Practicality.  The  feasibility  of  integrating  the  training 
technology  into  the  unit  without  interfering  with  mission  is  of  key 
concern.  Given  that  units  frequently  have  sparse  staffing  and 
limited  time  for  training,  it  is  important  that  any  new  technology 
serve  to  ease  the  training  requirement,  not  increase  time  demands. 
Thus,  developers  should  be  concerned  that  new  technologies  require 
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few  resources,  including  time,  manpower,  and  support.  Suggestions 
from  maintenance  personnel  in  the  field  included  the  notion  that 
training  tools  could  be  modular  in  format  to  allow  for  intermittent 
use  as  time  becomes  available,  and  that  technologies  should  not 
require  excessive  preparation  or  warm-up  time.  Minimal 
requirements  for  documentation  of  training  and  training  results  was 
a  key  concern  of  the  field.  Unnecessary  or  excessive  paperwork  is 
seen  as  a  major  detraction  to  the  practicality  of  any  proposed 
technology.  The  focus  should  be  on  the  conduct  of  training,  not 
its  documentation. 

Ratings  from  users  and/or  potential  users  would  supply 
information  relevant  to  practicality.  These  judgments  could  be 
obtained  by  supplying  SMEs  with  information  such  as  time  required 
for  training,  resources  requirements,  and  manpower  support 
requirements,  and  then  tasking  them  to  make  ratings  of  mission 
impact.  With  a  format  provided  by  Bierstedt,  Gillet,  and  Bentley 
(1989)  as  a  structure  for  practicality  ratings,  the  following  are 
examples  of  how  this  criterion  could  be  operationalized.  Managers 
would  be  presented  with  the  following  descriptions  which  they  would 
then  rate  with  the  following  mission  impact  scale.  Note  that  lower 
ratings  (i.e.,  1  or  2)  would  indicate  minimal  or  no  mission  impact 
and,  hence,  greater  practicality. 

1  *  Not  at  all 

2  -  To  a  small  extent 

3  *  To  a  moderate  extent 

4  =  To  a  great  extent 

5  ■  To  a  very  great  extent 

Example  1 : 

Given  a  training  time  of  20  hours  and  a  requirement  for 
portable  computers.  Technology  X  provides  unit-level  training 
on  Ancillary  Training  Y.  Will  this  Technology  X  negatively 
impact  the  mission? 


Example  2: 

Training  Technology  Z  will  train  3-skill  level  airmen  on 
troubleshooting  tasks  A,  B,  and  C.  Training  is  modular  in 
format,  typically  requiring  30-minute  sessions.  IVD  equipment 
is  required.  Will  Technology  Z  negatively  impact  the  mission? 


Example  3: 

Management  Technology  J  will  allow  shop  supervisors  to 
identify  key  tasks  to  serve  as  the  basis  for  OJT  during  the 
upgrade  period  to  the  5-skill  level.  This  technology  requires 
training  of  supervisors  (40  hours  in  FTD),  use  of  existing 
computers,  frequent  updates  from  MAJCOM,  and  monthly 
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documentation  and  reports.  Will  Technology  J  negatively 
impact  the  mission? 


Relevancy .  Users  are  concerned  that  any  time  and  technologies 
available  for  training  be  focused  on  critical,  mission  essential 
tasks.  The  training  technology  must  address  areas  of  need,  that 
is,  they  must  meet  field  requirements  and  not  expend  training 
resources  on  trivial  or  tangential  aspects  of  the  job.  An 
additional  requirement  is  that  the  technology  must  be  available  in 
a  timely  period;  technological  advancements  that  arrive  in  the 
field  after  their  time  frame  for  useful  contributions  has  passed 
will  be  seen  as  irrelevant  (e.g.,  "Why  couldn't  we  have  gotten  this 
two  years  ago  when  we  really  needed  itl"). 

Ratings  of  criticality  of  training  content  from  projected 
users  would  be  appropriate  to  address  this  issue.  In  a  manner 
comparable  to  the  Occupational  Measurement  Center's  format  of  the 
Occupation  Survey  Report  (OSR),  career  field  specialists  could 
provide  Likert-type  ratings  of  criticality,  training  emphasis,  task 
difficulty,  percent  performing,  and  so  on.  As  cited  in  the 
regulation  for  conducting  occupational  surveys,  ATCR  52-22  (1986), 
AL/HR  research  has  found  that  specialty  members  consider  factors 
such  as  percent  of  members  performing  a  task,  average  grade  levels 
of  personnel  performing  a  task,  and  task  criticality  when  making 
technical  training  emphasis  ratings. 

As  an  example  of  this  general  survey  approach,  the  maintenance 
data  collection  portion  of  the  SLT  project  gathered  training  criti¬ 
cality  ratings.  Each  interviewee  rated  a  series  of  ancillary  and 
job-related  tasks  using  the  following  scale: 

1  ■  Not  important 

2  =  Moderately  important 

3  =  Important 

4  ■  Very  important 

5  =  Extremely  important 

This  method  could  be  used  to  assess  SMEs'  judgments  of  the 
relevancy  of  training  content  for  any  technology.  It  may  even  be 
possible  to  access  existing  OSR  data  relevant  to  the  training 
content  of  each  technology. 

A  question  of  who  would  provide  the  most  valuable  task 
criticality  data  must  be  addressed  by  researchers,  and  it  may  be 
appropriate  to  tap  one  or  more  skill  levels  to  gather  comprehensive 
data.  Guidance  for  the  conduct  of  Air  Force  occupational  surveys 
describes  how  specialists  at  different  skill  levels  provide 
different  types  of  responses.  They  suggest,  for  example,  that 
while  9-skill  level  specialists  have  extended  work  experience,  they 
may  be  generally  unfamiliar  with  current  tasks.  The  best  sources 
of  job  information  are  typically  7 -skill  level  personnel  since  they 
possess  years  of  job  experience  and  perform  both  supervisory  and 
technical  tasks.  Airmen  at  the  5-skill  level  provide  the  best 
task-specific  information  within  a  work  center,  but  lack  career 
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ladder  knowledge.  Those  at  the  apprentice,  3-skill  level  are 
usually  highly  familiar  with  a  smaller  number  of  job  tasks  and  can 
provide  input  on  simple  or  repetitive  tasks  and  extra  work  details 
(e.g.,  work  area  clean-up,  preparation  for  inspections,  FOD  walk). 
Thus,  the  selection  of  SMEs  for  evaluation  of  training  technologies 
may  include  any  or  all  of  these  career  field  members,  dependent  on 
the  focus  of  the  planned  training. 

Flexibility.  An  additional  training  needs  requirement  that 
can  be  evaluated  is  how  well  the  training  tools  can  incorporate 
job-related  technological  updates  (e.g.,  changes  to  equipment 
and/or  procedures).  This  criterion  will  reflect  on  the  actual 
validity  of  the  technology;  that  is,  does  it  address  a  relevant 
concern  or  is  it  outdated  and  not  useable?  The  face  validity  of 
the  training  is  also  affected  since  trainees  are  unlikely  to  react 
favorably  or  attend  to  training  that  is  technically  out-of-date, 
even  when  the  basic  principles  are  still  valid.  Consequently,  the 
technology  must  be  capable  of  maintaining  credibility  and  currency 
through  updates  made  in  the  field  on  an  as-needed  basis.  Training 
technology  developers  can  evaluate  this  aspect  of  the  methodology 
and  provide  data  to  be  considered  by  other  evaluators.  That  is, 
the  flexibility  of  the  system  or  tool  is  best  determined  by  the 
developers,  although  user  input  is  also  essential. 

Components  of  the  flexibility/ "updateability"  criterion 
could  serve  as  the  basis  for  a  multiple-item  assessment. 
Dimensions  to  be  considered  include:  (a)  requirements  for 
technical  task-specific  data,  documentation,  technical  orders, 
and  so  on;  (b)  format  of  these  data  (e.g.,  text,  computer 
diskette);  (c)  methods  and  requirements  for  incorporating  new 
data  into  the  system;  (d)  expected  frequency  of  future  updates 
and  changes;  and  (e)  personnel  requirements  for  making  updates. 

Reliability  and  Maintainability.  Essential  "ingredients"  of 
a  unit-level  training  technology  include  that  it  be  reliable  (i.e., 
works  on  a  regular  basis  with  little  or  no  error)  and  require 
little  or  no  maintenance  (i.e.,  can  withstand  use  and  misuse  of 
unit-level  trainers  and  trainees).  This  criterion  is  primarily 
applicable  for  those  training  technologies  that  require  extensive 
use  of  computer  hardware /software  or  equipment.  Dependable 
technologies  are  essential  for  unit-level  users  since  it  is  not 
feasible  for  any  central  authority  to  monitor  and  maintain  the 
technology;  the  users  must  be  able  to  do  this.  Any  technology  that 
requires  frequent  and/or  extensive  maintenance  in  the  field  is 
unlikely  to  be  effective  or  efficient  since  unit  training  resources 
should  be  devoted  to  training  and  not  maintenance  of  technologies. 
While  this  is  listed  under  "User  Acceptability,"  developers  should 
be  able  to  provide  test  data  and  information  relevant  to  this 
criterion.  The  ultimate  tests  are,  however,  actual  user  reports  and 
empirical  field  data  such  as  the  reliability  (Mean  Time  Between 
Failures  [MTBF] )  and  maintainability  (Mean  Time  to  Repair  [MTTR] ) 
figures  currently  reported  in  monthly  base  level  maintenance 
reports . 

Enhances  Hands-on  Training.  Consideration  of  this  aspect  of 
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training  technologies  is  somewhat  related  to  the  issues  of 
relevancy.  In  this  case,  however,  we  are  looking  at  relevancy  of 
the  purpose  of  training  ( i . e . ,  complementing  OJT )  rather  than  the 
content  of  training  (i.e.,  task  criticality).  Comments  elicited  by 
the  SLT  interview  protocol  indicated  general  concern  at  the  unit 
level  for  technologies  that  will  enhance  hands-on  training  and  be 
a  complement  to  OJT.  Hands-on  training  is  seen  by  field  personnel 
(e.g.,  OJT  trainers,  shop  supervisors)  as  being  the  most  valuable 
approach  to  insuring  task  competency  and  unit-level  personnel  are 
interested  in  improving  its  approach,  making  it  a  useful  criterion 
for  distinguishing  between  technologies.  Evaluations  by  training 
technology  developers  can  address  the  applicability  of  new  tools  to 
hands-on  training. 

Management  Acceptability 

For  purposes  of  this  report,  we  are  defining  "management”  as 
those  decision-makers  above  the  unit  level  who  are  responsible  for 
selecting,  evaluating,  and  funding  training  technologies.  It  is 
important  to  consider  functional  managers  and  training  system 
managers  at  all  levels  since  different  viewpoints  can  provide  valid 
information  related  to  the  criteria  discussed  below.  For  example, 
personnel  at  the  base/wing.  Numbered  Air  Force  (NAF),  MAJCOM,  and 
Headquarters  USAF  could  be  included  as  SMEs.  Some  of  the  criteria 
included  in  this  general  area  of  "Management  Acceptability"  are 
parallel  to  "Unit-Level  User  Acceptability,"  however,  the  content 
of  the  data  is  different,  reflecting  unique  information  relevant 
for  evaluating  the  training  technologies. 

Criticality  of  Need.  Managers  have  a  broad  view  of  mission 
requirements  and  should  be  able  to  provide  input  on  current  and 
projected  training  needs.  They  can  make  evaluations  of  the 
criticality  of  the  training  addressed  by  each  training  technology. 
They  can  also  evaluate  the  appropriateness  of  the  time  frame  of 
technology  development  and  assess  whether  the  scheduled  development 
milestones  will  adequately  meet  training  needs. 

Practicality .  The  issue  of  practicality  is  also  a  relevant 
topic  and  concern  for  managers  at  various  levels  (e.g.,  base, 
MAJCOM,  NAF,  HQUSAF).  They  should  be  able  to  assess  whether  the 
technologies  can  be  integrated  into  the  unit  without  interfering 
with  mission  accomplishment.  At  a  more  global  level  (i.e.,  above 
that  of  the  unit  level),  managers  can  also  provide  input  as  to  the 
practicality  and  mission  impact  of  the  resource  requirement  data 
supplied  by  developers  and  program  managers  for  each  technology 
(e.g.,  funding,  staffing,  time). 

Degree  of  Advancement.  Perhaps  the  single-most  important 
criterion  relevant  to  the  potential  of  a  new  training  technology  is 
its  degree  of  advance  or  anticipated  gain.  Decision-makers  will 
need  to  answer  the  question:  Does  the  technology  provide  a 
detectable  and  meaningful  improvement  over  existing  capabilities? 
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This  is  the  strict  interpretation  of  the  concept  of  "utility,"  that 
is,  the  extent  to  which  a  new  technology  improves  upon  existing 
capabilities.  In  this  case,  enhanced  trainee  performance  over  the 
current  level  is  the  most  appropriate  measure  of  gain.  Various 
operational  measures  of  performance  include  time  to  upgrade. 
Enlisted  Performance  Ratings,  mission  achievement  measures  (e.g., 
number  of  sorties,  in  commission  rates,  maintenance  productivity), 
and  so  on.  However,  these  may  be  insufficient  to  detect  changes  in 
individual  performance  level  and  it  is  possible  that  new  measures 
of  performance  and  training  achievement  would  be  needed.  Archival 
records  and  research  by  developers  can  provide  data  relevant  to 
this  criterion,  although  decision-makers  will  have  to  ultimately 
make  value  judgments  as  to  the  extent  of  the  anticipated  gain. 

Degree  of  Application.  Each  technology  must  be  evaluated  to 
determine  if  it  provides  a  means  of  improving  capabilities  across 
a  wide  field  of  training  applications  or  whether  it  is  filling  a 
specific  perceived  need.  Limited  application  may  be  appropriate  if 
the  need  is  urgent  and  the  technology  provides  a  meaningful 
solution.  However,  a  more  generalizable  technology,  appropriate 
across  numerous  training  needs  may  be  viewed  as  advantageous  for 
most  concerns.  Generalizability  of  application  of  a  technology  may 
be  characterized  by  the  number  and  diversity  of:  (a)  AFSCs  for 
which  the  training  i<3  designated;  (b)  skill  levels  for  which 
training  is  appropriate;  (c)  work  centers  within  units  that  can 
utilize  the  training;  (d)  tasks  included  in  the  training;  and/or 
(e)  weapon  systems  and/or  mission  areas  appropriate  for 
implementation  of  training.  Training  developers  can  supply 
information  relevant  to  the  application  of  the  technology, 
although  user  and  management  input  may  also  provide  valid  data  on 
the  degree  of  application.  All  of  these  data  could  feed  into  an 
overall  evaluation  of  this  criterion. 

Reliability  and  Maintainability.  The  reliability  and 
maintainability  of  technologies  is  a  contributor  to  management 
acceptability,  as  it  was  relevant  for  user  acceptability.  The  same 
issues  discussed  previously  are  pertinent  here,  too,  although 
managers  would  consider  the  impact  of  this  criterion  across  a 
number  of  units  and/or  applications.  Technical  support 
requirements  and  test  data  from  developers  (e.g.,  MTBF,  MTTR)  and 
ratings  from  users  on  this  criterion  could  be  used  as  input  for 
managers  in  making  overall  evaluations  of 
reliability /maintainability . 

Technology  Base.  Ideally,  any  effort  selected  for  development 
into  a  working  system  will  have  been  sufficiently  studied  to  ensure 
that  a  high  confidence  exists  that  will  enable  the  developing 
organization  to  minimize  the  risk  of  application.  Thus,  a  "broad 
base"  technology  would  have  several  to  many  studies  with  positive 
results  indicating  that  there  is  a  strong  potential  for  successful 
implementation  into  a  typical  Air  Force  unit.  A  technology  with  a 
riskier,  more  "narrow"  base  may  be  seen  as  being  advantageous  if 
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the  potential  for  advancement  is  great. 

This  criterion  is  not  independent  from  those  described 
earlier,  instead,  it  serves  as  a  type  of  "summary"  which  reflects 
the  results  of  preliminary  study  and  evaluation.  Specific  studies 
that  contribute  to  the  advancement  of  the  technology  base  include: 
criticality  of  training  need,  degree  of  application,  timeliness  of 
implementation,  under standability,  user  acceptability,  reliability, 
and  so  on.  Those  technologies  which  have  been  thoroughly 
researched  with  positive  results  in  several  of  these  areas  are  more 
likely  to  be  successfully  implemented,  and  therefore  more  worthy  of 
continued  R&D,  than  those  technologies  which  have  little  or  no 
supporting  evidence  or  research  findings.  Technology  developers 
could  assemble  relevant  study  data  for  categorization  by  evaluators 
to  describe  the  breadth  of  the  technology  base. 

Economy /Cost .  A  final  consideration  relevant  to  the 
"Management  Acceptability"  concept  is  the  issue  of  cost  of 
technologies.  At  some  point  in  any  developmental  effort  the 
funding  requirements  become  a  deciding  factor.  It  has  been 
presented  here  last,  not  to  minimize  the  recognition  of  its 
importance,  but  to  rather  highlight  the  emphasis  that  should  be 
given  on  the  other  non-cost  considerations. 

The  ultimate  decision  point  for  those  trying  to  prioritize 
training  technology  R&D  efforts  is:  Will  it  provide  sufficient 
advancement  to  justify  the  cost  of  development?  Thus,  the 
"benefits"  of  the  training  technology  (e.g.,  improved  job 
performance,  reduced  training  time,  improved  training 
effectiveness,  greater  user  satisfaction,  improved  mission 
readiness)  must  be  somehow  balanced  against  the  "cost." 

The  issue  of  how  "cost"  and  "economy"  considerations  are 
incorporated  into  a  decision  scenario  is  critical.  Dollar  costs 
must  be  presented  in  a  metric  different  than  "benefits"  since  they 
are  not  necessarily  equatable  (Colella,  1986).  As  an  example, 
identified  costs  (e.g.,  cost  per  unit  of  training/module,  cost  per 
training  hour,  expense  of  equipment,  manning  requirements  for 
support  of  technology)  cannot  be  directly  subtracted  from  accrued 
benefits  (e.g.,  increased  training  efficiency,  increased  job 
performance,  shorter  time  to  upgrade,  enhanced  readiness).  Colella 
also  argued  that  since  military  goals  are  usually  stated  in  terms 
of  combat  readiness  and  not  a  dollar  metric,  an  individual's 
"worth"  is  best  described  by  their  contribution  to  the  mission  and 
not  as  a  cost  to  the  organization.  Researchers  will  have  to 
identify  the  most  appropriate  "benefits"  and  "costs"  to  assess  for 
the  purposes  of  evaluating  the  training  technologies.  A  common 
metric,  however,  across  technologies  must  be  used  to  validly  assess 
differences  among  technologies. 

The  assessment  of  the  cost  effectiveness  of  training  and 
training  technologies  is  an  issue  of  great  concern  for  researchers 
and  decision-makers.  Much  research  has  been  advanced  within  the 
military  in  efforts  to  quantify  the  inputs  and  outcomes  of  training 
expenditures  and  the  AL/HRTE  may  find  guidance  within  work 
previously  done  in  the  area.  For  example,  Hammon  and  Horowitz 
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(1989)  provided  data  that  support  the  contention  that  "quantitative 
relationships  that  support  the  proposition  that  more  flying  results 
in  measurably  better  performar.ee  have  been  developed  for  the  Air 
Force  and  Navy”  (p.  1).  These  quantitative  relationships  include 
connections  between  flying  training  and  better  landing  attempts 
aboard  aircraft  carriers  and  greater  bombing  accuracy.  Examples  of 
other  relevant  studies  include  research  on  automation  of  classroom 
instruction  (Thoreson,  1988),  alternative  aircrew  devices  (Marcus, 
Patterson,  Bennett,  &  Gershan,  1980),  analyses  of  individual  skill 
training  (Solomon,  1986),  and  use  of  maintenance  data  as  measures 
of  performance  (String  &  Orlansky,  1981). 

Necessary  for  evaluation  is  the  specification  of  all  costs 
related  to  a  technology  so  that  valid  comparisons  and  trade-offs 
can  be  made.  Knapp  and  Orlansky  (1983)  provide  guidance  for  the 
a3sessment  of  the  life-cycle  costs  of  any  course,  program,  or 
training  technology.  In  this  report,  three  cost  categories  are 
listed  which  are  useful  for  describing  the  technology  life-cycle 
(i.e.,  research/development,  initial  investment, 
operations /support)  in  relation  to  three  types  of  training  (i.e., 
flight  simulators,  computer-based  training,  maintenance 
simulators ) .  Additional  guidance  may  be  found  in  research  by  the 
Marines  (cf.,  Patterson  &  Adelman,  1981)  and  the  Army  (cf., 
Matlick,  Berger,  &  Rosen,  1980). 

C.  Assumptions  for  Application  and 
Use  of  Evaluative  Criteria 

The  following  is  a  list  of  assumptions  and  guidelines  for 
research  on  the  evaluation  of  the  training  technologies.  These 
serve  to  summarize  the  general  process  of  applying  evaluative 
criteria  in  the  decision-making  process. 

1.  Data  will  be  gathered  from  multiple  sources  (e.g.,  unit- 
level  users,  managers  at  the  unit,  wing,  base,  MAJCOM, 
HQUSAF  levels ) . 

2 .  Multiple  sources  may  be  appropriate  for  some  criteria 
while  other  criterion  evaluations  are  appropriate  from  a 
single  source. 

3.  A  single  set  of  criteria  will  be  applied  across  all  tech¬ 

nologies  to  insure  a  standard  upon  which  to  base 
decisions.  It  is  important  the  evaluators  do  not 
selectively  include  some  criteria  while  omitting  others. 

4.  Prior  to  any  large-scale  effort  to  systematically  assess 
technologies,  each  measure  must  be  examined  for 
usefulness  and  reliability  (i.e.,  validated). 

5.  Weighting  schemes  will  be  required  for  combining  data 
across  sources  and  criteria.  A  common  metric  must  be 
used  for  each  criterion  prior  to  any  combining  of  data. 

6.  Application  of  these  evaluative  criteria  can  be  used  in 
the  formulation  of  an  S&T  Investment  Strategy;  analyses 
will  yield  information  which  can  be  used  by 
decision-makers  for  prioritization  of  training  technology 
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R&D  efforts. 


D.  Benefits  of  Enhanced  SLT 

One  of  the  major  criteria  identified  in  this  paper  for 
evaluating  potential  technology  applications  was  the  cost-benefit 
tradeoff.  We  felt  that  it  would  be  helpful  at  this  point  in  the 
paper  to  identify  some  of  the  potential  benefits  associated  with 
training  technology  applications  and  enhanced  SLT. 

Increased  training  activity  at  the  squadron  level  has  been 
thought  to  have  potential  economic  advantage  for  the  Air  Force. 
Economies  in  cost  of  formal  training  (i.e.,  pipeline  and  TDY  to 
school  funds)  are,  perhaps,  the  most  frequently  cited  economic 
outcome  of  SLT.  Other  possible  outcomes  of  SLT  include  the 
benefits  associated  with  enhanced  effectiveness  and  efficiency  of 
training  such  that  trainees  become  better  and  more  quickly  equipped 
to  fulfill  job  requirements.  This  increased  training  effectiveness 
could  lead  to  improved  job  performance,  increased  productivity,  and 
overall  enhanced  mission  accomplishment. 

At  this  point  in  the  SLT  Research  Initiative,  it  is  premature 
to  declare  that  training  at  the  squadron  would  result  in  economic 
benefits.  Nor  can  specific  HSD  training  technologies  be  singled 
out  for  accelerated  R&D  and  implementation  based  wholly  or  partly 
on  cost  savings  considerations.  Howevar,  it  is  possible  to 
identify  the  types  of  outcomes  that  the  technologies  are  purported 
to  produce  in  relation  to  their  application  to  SLT.  Summary  data 
supplied  by  training  technology  managers  in  response  to  the  AL/HR 
request  for  information  served  as  the  primary  input  for  this 
discussion  of  possible  enhancements  to  and  benefits  of  SLT  (see 
Appendix  E  for  brief  descriptions  of  HSD  technologies  and  Appendix 
F  for  the  AL/HR  request  for  information). 

The  program  managers'  responses  were  reviewed  and  the 
following  anticipated  benefits  extracted.  These  specific 

anticipated  outcomes  have  been  grouped  under  general  categories  of 
Training  Efficiency,  Quality  of  Training,  Associated  Costs,  Manning 
and  Staffing  Issues,  and  Performance  Outcomes.  Below  is  a  listing 
of  these  expected  outcomes  and  the  relevant  HSD  training 
technologies.  (Note.  Not  all  of  the  training  technologies  are 
represented  in  the  following  list;  only  those  most  appropriate  for 
application  to  maintenance  units  were  reviewed.  These  are  listed 
in  no  particular  order.) 

Training  Efficiency 

1.  Reduce  time  required  for  development  of  training  [AIDA, 
ATS,  ITT]. 

2.  Decrease  number  of  forms  and  required  paperwork  [AOTS]. 

3.  Centralize  and  organize  training  data  [AOTS]. 

4.  Utilize  slack  machine  time  for  training  [Embedded 
Training] . 

5.  Faster  and  better  initial  training,  less  retraining 
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6. 


required  [ ITT ] . 

Replaces  less  efficient  technologies  (e.g., 
transparencies  and  grease  pencils)  [ITT]. 

7 .  Eliminates  training  requirements  for  certain  tasks 
[ JATAT ] . 

8.  Decreases  training  time  [Part-Task  Training]. 

9.  Reduces  need  for  residential  training  [Distance 
Learning] . 

10.  Allows  transport  of  locally  developed  computer-based 
training  (CBT)  to  other  squadrons  [CBT  Selection  and 
Implementation  Strategies ] . 

11.  Allows  for  more  efficiently  structured  and  cost  effective 
manpower,  personnel,  and  training  programs  [TDMT]. 

Quality  of  Training 

1.  Provide  consistent,  high  quality  computer-based  training 

[AIDA,  ISS,  CBT  Selection  and  Implementation  Strategies]; 
reduce  inappropriate  development  and  use  of  CBT  [CBT 
Selection  and  Implementation  Strategies ] . 

2.  Improve  training  quality  and  realism  [ATS]. 

3.  Provide  training  problem  and  deficiency  diagnosis  [AOTS]. 

4.  Promote  standardization  and  increased  control  of  OJT 
[AOTS] . 

5.  Simulation  of  hands-on  experience  not  otherwise  available 

[ITT]. 

6.  Allows  more  opportunities  for  individual  student  - 
instructor  interaction  [ITT]. 

7.  Enhances  classroom  instruction  [ICATT]. 

8.  Enhances  FTD  [Distance  Learning]. 

Associated  Costs 


1.  Reduce  TDY  funding  for  school  attendance  [AOTS,  BJS, 
Embedded  Training,  Distance  Learning]. 

2.  Reduction  in  classroom  instruction  time  (20-40%)  [ICATT]. 

3.  Provides  ability  to  do  costing  of  OJT  [TDMT]. 

4.  Identification  of  the  impacts  of  training  decisions  on 
costs,  capacities,  and  resource  requirements  [TDMT]. 

Manning  and  Staffing  Issues 

1.  Increased  availability  of  personnel  [AOTS]. 

2.  Utilize  slack  personnel  time  for  training  [Embedded 
Training] . 

3.  Decrease  washback, washout  rates  (40-80%)  [ICATT]. 

4.  Portability  of  technology  allows  for  on-site 
training/assistance  [ IMIS ] . 

5.  Provides  a  mechanism  for  career  field  training  management 

and  planning  [TDMT]. 

Performance  Outcomes 
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1.  Increased  readiness  and/or  mission  accomplishment  [AOTS, 
BJS  ] . 

2.  Accelerated  acquisition  of  complex  technical  skills 
[BJS]. 

3.  Increased  adaptability  of  technicians  and  broadening  of 
job  responsibility  [BJS]. 

4.  Increase  shop  productivity  [BJS]. 

5.  Increase  student  proficiency  up  to  40%  [ICATT]. 

Examination  of  these  reveals  a  general  lack  of  specificity  in 
the  description  of  the  benefits.  This  lack  of  precision  is  most 
likely  an  indication  of  the  incomplete  state  of  development  of  each 
technology ,  that  is,  they  are  in  various  development  stages  and 
have  not  been  implemented,  nor  have  outcomes  been  measured.  The 
vagueness  of  the  descriptions  also  reflects  the  difficulty  of 
quantification  of  results  which  is  inherent  in  training  evaluation 
studies . 

Nevertheless,  the  program  managers  of  the  technologies  cited 
above  indicated  that  their  respective  programs  had  implications  for 
enhancement  of  SLT.  Further  R&D  of  the  technologies  and  continued 
SLT  research  should  focus  on  the  application  of  the  technologies  to 
the  training  needs  at  the  unit  level. 


